<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:27:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31457] Mongod stop responding, takes 200 load and don&apos;t even switch to secondary</title>
                <link>https://jira.mongodb.org/browse/SERVER-31457</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;I just suffered a quite bad issue with mongod 3.4.5 (WT), the requests were totally normal, nothing out of the ordinary and suddenly it started taking up to 200 load on my server, and all 8 CPUs of course:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/167583/167583_mongodb-incident-1.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/167582/167582_mongodb-incident-2.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;At this point the server stopped responding to any request, but it seems it kept pinging the secondaries and syncing as it stayed primary until I manually changed the priory from the secondary (I couldn&apos;t even SSH on the primary as it was killing the machine).&lt;/p&gt;

&lt;p&gt;As I had numerous problem of the kind in the past due tu various performance issues in WT, cache eviction, etc. (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700&quot; title=&quot;WT secondary performance drops to near-zero with cache full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27700&quot;&gt;&lt;del&gt;SERVER-27700&lt;/del&gt;&lt;/a&gt;) I tried to let it rest see if it recovers but after 3 hours had to hard reboot the server to get it back...&lt;/p&gt;

&lt;p&gt;I checked the logs after the reboot and there was just no single line of log during the 3 hours, and the ones before the crash have nothing weird to me. I collected the diagnostic dir, I can give it to you (and the last hour of logs) if you send me your usual upload link.&lt;/p&gt;

&lt;p&gt;If you can access my MongoDB Cloud Manager stats, the project id is: 5012a0ac87d1d86fa8c22e64 otherwise I can give you some screenshots, but there&apos;s nothing very interesting as these charts were all totally normal until the agent stopped collecting data.&lt;/p&gt;

&lt;p&gt;Thanks for your help&lt;/p&gt;</description>
                <environment></environment>
        <key id="442150">SERVER-31457</key>
            <summary>Mongod stop responding, takes 200 load and don&apos;t even switch to secondary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="bruce.lucas@mongodb.com">Bruce Lucas</assignee>
                                    <reporter username="bigbourin@gmail.com">Adrien Jarthon</reporter>
                        <labels>
                            <label>SWNA</label>
                    </labels>
                <created>Sun, 8 Oct 2017 08:45:57 +0000</created>
                <updated>Tue, 7 Nov 2017 20:10:47 +0000</updated>
                            <resolved>Tue, 10 Oct 2017 17:08:53 +0000</resolved>
                                    <version>3.4.5</version>
                                                    <component>Stability</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1694802" author="bruce.lucas@10gen.com" created="Tue, 10 Oct 2017 17:08:53 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;OK, I&apos;ll close this ticket as you requested. Sorry we weren&apos;t able to diagnose the problem, and please do let us know if it occurs again.&lt;/p&gt;

&lt;p&gt;Bruce&lt;/p&gt;</comment>
                            <comment id="1693850" author="bigbourin@gmail.com" created="Mon, 9 Oct 2017 21:57:46 +0000"  >&lt;p&gt;Ok great, at least I&apos;m not the first one to get a probably similar issue (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-14139&quot; title=&quot;Disk failure on one node can (eventually) block a whole cluster&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-14139&quot;&gt;&lt;del&gt;SERVER-14139&lt;/del&gt;&lt;/a&gt;) and some measure have already been taken (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29947&quot; title=&quot;Implement Storage Node Watchdog&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29947&quot;&gt;&lt;del&gt;SERVER-29947&lt;/del&gt;&lt;/a&gt;) &#9786;&lt;/p&gt;

&lt;p&gt;Unfortunately this feature is for mongodb enterprise so I can&apos;t use it ^^ it&apos;s sad to think most clusters can die on a - not so uncommon - IO failure but I understand some efforts must go toward financing for the future of mongodb &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; Though it&apos;s definitely not making me more willing to pay for mongodb someday if I can&apos;t trust it.&lt;/p&gt;

&lt;p&gt;I&apos;ll start watching &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29980&quot; title=&quot;Built-in hang detection diagnostics and recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29980&quot;&gt;&lt;del&gt;SERVER-29980&lt;/del&gt;&lt;/a&gt; as this could improve the overall stability of the system and would also have helped with software issues I encountered like &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700&quot; title=&quot;WT secondary performance drops to near-zero with cache full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27700&quot;&gt;&lt;del&gt;SERVER-27700&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks for the gdb command, I would have tried this of course (and strace, etc..) if only I had access to the machine, yes. You can close this ticket then, I don&apos;t expect this issue again anytime soon as it was running for couple years on this server. But if I ever manage to get more details I&apos;ll let you know.&lt;/p&gt;</comment>
                            <comment id="1693174" author="bruce.lucas@10gen.com" created="Mon, 9 Oct 2017 16:05:40 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;Thanks for the additional info. Regarding your question &quot;where&apos;s the separation between system and iotime&quot;, I believe the graph labeled &quot;i/o&quot; probaby refers to the CPU iowait statistic that I mentioned above, and would (if present) mean that the CPUs are not busy because they are waiting for i/o.&lt;/p&gt;

&lt;p&gt;Instead we see a high utilization of system (kernel) CPU time. Generally I would not expect this to be i/o as such, although it is possible that it is an issue in mongod that is triggered by an i/o problem. We have seen high kernel utilization by mongod in some cases in the past, such as &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3345&quot; title=&quot;Improve rwlock scaling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3345&quot;&gt;&lt;del&gt;WT-3345&lt;/del&gt;&lt;/a&gt;, where a WiredTiger was doing excessive spinning while waiting for something. That particular issue is fixed in the version you are running, but it&apos;s possible there is an additional undiagnosed issue of this type.&lt;/p&gt;

&lt;p&gt;Regarding robustness against storage issues, we recently introduced a new storage watchdog feature that would allow the node to detect a storage problem and fail over to another node; see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29947&quot; title=&quot;Implement Storage Node Watchdog&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29947&quot;&gt;&lt;del&gt;SERVER-29947&lt;/del&gt;&lt;/a&gt; for details. You might wish to enable this if you suspect this is a storage issue and it could occur again.&lt;/p&gt;

&lt;p&gt;We also have an open ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29980&quot; title=&quot;Built-in hang detection diagnostics and recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29980&quot;&gt;&lt;del&gt;SERVER-29980&lt;/del&gt;&lt;/a&gt; to consider how to provide additional robustness by detecting hangs of any kind, but that is a much larger undertaking, not the least of which is how to define and detect a &quot;hang&quot;.&lt;/p&gt;

&lt;p&gt;Finally, if this were to occur again it would be very valuable to have stack traces from the mongod process, using the following command:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gdb -ex &quot;thread apply all bt&quot; --batch --pid $(pidof mongod) &amp;gt;stacks.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;I realize however this could be problematic if there is no way to ssh to the machine during the incident.&lt;/p&gt;

&lt;p&gt;With the information we have available at this time I don&apos;t see any way forward with a diagnosis; if you would like we can leave this ticket open in &quot;waiting for user input&quot; state so that you can provide whatever additional information you can collect if there is a re-occurrence.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1693102" author="bigbourin@gmail.com" created="Mon, 9 Oct 2017 15:24:18 +0000"  >&lt;p&gt;I check the other charts I have from nixstats (which managed to monitor the machine during most of the time), here is the CPU time:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/167627/167627_cpu-time.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;br/&gt;
seems to be system time, not iotime. But i&apos;m not sure where&apos;s the separation between system and iotime as io can stem from a lof of different syscalls.&lt;/p&gt;

&lt;p&gt;We can also see the disk usage which dropped to &lt;b&gt;almost&lt;/b&gt; 0 (the chart shows a couple writes / sec if I hover it) but only mongod uses it so it makes sense.&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/167628/167628_disk-usage.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I agree it&apos;s weird that I couldn&apos;t access ssh or else during the incident, and I find very strange that not a single application was able to log anything whereas nixstat was still reporting data. It looks to me as if there was an IO error on the machine indeed, and maybe sshd blocked on some log writing when I tried to connect. nixstats may have kept working because it&apos;s not writing a single line to disk, atop for example (which I trust to be much more robust than nixstats) didn&apos;t log any data after the incident, and when I try to navigate to the time of the incident in the history file, it crashes &#9785; so most probably corrupted file.&lt;/p&gt;

&lt;p&gt;I can&apos;t check much with my system provider, it&apos;s a dedicated machine and it&apos;s working fine now so they won&apos;t find anything.&lt;br/&gt;
I&apos;ll try to see if I can find any way to detect this kind of issue with more accuracy.&lt;/p&gt;

&lt;p&gt;In the meanwhile, there&apos;s still something not very pleasant about the way mongod behaved faced with this issue, as it kept pinging the secondary which prevented the failover whereas it was totally stuck for more than 10 minutes. Though I&apos;m not sure how to improve this, it sounds a bit silly to force and IO operation on ping just to make sure IO is working properly.&lt;/p&gt;</comment>
                            <comment id="1692935" author="bruce.lucas@10gen.com" created="Mon, 9 Oct 2017 14:18:55 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;Thanks for uploading the data. Unfortunately we are not finding any information about what went wrong - the data looks normal right up until the point where data collection stops, just as you noted with the Cloud metrics.&lt;/p&gt;

&lt;p&gt;It seems unusual that you were completely unable to ssh to the machine, even if it is under heavy load. Note for example that the Ruby process was still able to get a moderate portion of the CPU, so it was not completely starved out by the mongod process. Can you say more about what it was doing during this time? This makes it seem unlikely that that sshd was unresponsive just because of heavy CPU load.&lt;/p&gt;

&lt;p&gt;Is it possible that the reported mongod CPU utilization was actually iowait? This represents time that a CPU could not be used because threads were waiting for i/o. It is reported as a CPU time, but it is actually a kind of idle time. I believe threads waiting for i/o also count towards the load average. There&apos;s a good article on the subject &lt;a href=&quot;https://prutser.wordpress.com/2012/05/05/understanding-linux-load-average-part-2/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. Do you have available metrics that break out CPU utilization by user, kernel, iowait, etc.?&lt;/p&gt;

&lt;p&gt;If there were a system-level storage problem that could possibly affect mongod in a way consistent with the data, and also prevent sshd from accepting connections. Can you check with your service provider about this?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1692432" author="bruce.lucas@10gen.com" created="Sun, 8 Oct 2017 14:22:43 +0000"  >&lt;p&gt;Thanks Adrien, we will take a look at the data.&lt;/p&gt;</comment>
                            <comment id="1692422" author="bigbourin@gmail.com" created="Sun, 8 Oct 2017 13:44:59 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;Sure, I uploaded:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The mongodb log for the hour of incident (19h to last written line at 19h50)&lt;/li&gt;
	&lt;li&gt;The syslog file for the same period + the next line 4.5 hours later&lt;/li&gt;
	&lt;li&gt;The diagnostic.data archive&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The charts are in CEST (GMT+2) timezone, so the incident started at 5:50 pm UTC and the hardware reboot occurred at 10:22 pm.&lt;br/&gt;
Well it&apos;s the first time I encounter this specific but I had a similar one a few months back, where mongod would put my machine at 1000 load (yep) but without actually using the CPU (&amp;lt; 1 core used) this was very weird, as if 1000 threads were waiting on something but not consuming ressources, I&apos;m not even sure how it&apos;s possible. And this time it properly failed over to secondary, I restarted the process and all was good, in the log file I saw some kind of critical exception (segfault or similar) but only when I stopped the server not when the problem started. I didn&apos;t report the issue back then because I deleted the data dir to re-sync the server and forgot about copying the diagnostic.data dir so it would have been hard for you to diagnose.&lt;/p&gt;

&lt;p&gt;About the upgrades I upgraded the primary to 3.4.9 tonight after the incident but otherwise it was running 3.4.5 since June 20th, and it was previously on 3.4.4 since May 20th.&lt;/p&gt;

&lt;p&gt;Let me know if there&apos;s anything else I can help you with !&lt;/p&gt;</comment>
                            <comment id="1692406" author="bruce.lucas@10gen.com" created="Sun, 8 Oct 2017 12:58:56 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;Thanks for the report, and sorry that you encountered this problem.&lt;/p&gt;

&lt;p&gt;Can you please upload an archive of diagnostic.data and (if possible) the entire mongod log file covering the time of the issue &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/2d905051-0a91-4cce-9a80-2575b19eb6a0.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Also can you please tell us&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;What is the timezone of the times in the charts that you posted?&lt;/li&gt;
	&lt;li&gt;Is this the first time you&apos;ve encountered this specific issue?&lt;/li&gt;
	&lt;li&gt;What is the recent mongod upgrade history of this node?&lt;/li&gt;
	&lt;li&gt;Are there any messages in syslog (/var/log/messages or /var/log/syslog) around the time of the incident?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;

</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="400801">SERVER-29980</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="399792">SERVER-29947</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="167627" name="cpu-time.png" size="22439" author="bigbourin@gmail.com" created="Mon, 9 Oct 2017 14:37:38 +0000"/>
                            <attachment id="167628" name="disk-usage.png" size="27978" author="bigbourin@gmail.com" created="Mon, 9 Oct 2017 14:39:35 +0000"/>
                            <attachment id="167583" name="mongodb-incident-1.png" size="24478" author="bigbourin@gmail.com" created="Sun, 8 Oct 2017 08:33:11 +0000"/>
                            <attachment id="167582" name="mongodb-incident-2.png" size="20943" author="bigbourin@gmail.com" created="Sun, 8 Oct 2017 08:33:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 8 Oct 2017 12:58:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 18 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 18 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bigbourin@gmail.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hthb0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht97dr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Impossible to know of course&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htgx47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>