<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:31:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53482] Invariant failure: opCtx != nullptr &amp;&amp; _opCtx == nullptr in src/mongo/db/client.cpp, line: 126</title>
                <link>https://jira.mongodb.org/browse/SERVER-53482</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I&apos;ve seen the above on three different occasions since upgrading to to 4.4.x.&lt;/p&gt;

&lt;p&gt;The first time was on 4.4.1.&#160; I was adding a hidden node to the replica set.&#160; The primary locked up for a few seconds and then all three nodes crashed with the same fatal error.&lt;/p&gt;

&lt;p&gt;The second time was on 4.4.x and I don&apos;t recall anything special going on.&#160; It only crashed one instance that time.&lt;/p&gt;

&lt;p&gt;The third time was on 4.4.2-ent (Cloud Manager Backup limited license) and it happened unprovoked.&#160; It started with the primary locking up for about 10 seconds (no log activity, failed health check) and then the remainder of the servers crashed as soon as they were elected primary.&lt;/p&gt;

&lt;p&gt;In all three cases, the ctx in the log was:&#160;TopologyVersionObserver and the most recent case also had logs with ctx=monitoring-keys-for-HMAC.&lt;/p&gt;

&lt;p&gt;I could not find any similar issues regarding this particular invariant failure (client.cpp).&lt;/p&gt;

&lt;p&gt;Happy to provide additional context/logs if you feel this is worth investigating, otherwise I will likely have to revert to 4.2.&#160; The above invariant failure has only happened on 3 occasions, but the mongod freeze they started with has happened over a dozen times in the last 2 months.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1576335">SERVER-53482</key>
            <summary>Invariant failure: opCtx != nullptr &amp;&amp; _opCtx == nullptr in src/mongo/db/client.cpp, line: 126</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="b.granetzke@fetchrewards.com">Brian Granetzke</reporter>
                        <labels>
                    </labels>
                <created>Tue, 22 Dec 2020 20:22:34 +0000</created>
                <updated>Wed, 27 Jan 2021 14:49:07 +0000</updated>
                            <resolved>Wed, 27 Jan 2021 14:33:51 +0000</resolved>
                                    <version>4.4.1</version>
                    <version>4.4.2</version>
                                                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="3584806" author="dmitry.agranat" created="Wed, 27 Jan 2021 14:33:38 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;, as this issue is now being investigated in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52735&quot; title=&quot;mongodb crash with &amp;quot;Invariant failure&amp;quot; opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52735&quot;&gt;&lt;del&gt;SERVER-52735&lt;/del&gt;&lt;/a&gt;, I will go ahead and close this case. Please watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52735&quot; title=&quot;mongodb crash with &amp;quot;Invariant failure&amp;quot; opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52735&quot;&gt;&lt;del&gt;SERVER-52735&lt;/del&gt;&lt;/a&gt; for updates.&lt;/p&gt;</comment>
                            <comment id="3575622" author="b.granetzke@fetchrewards.com" created="Thu, 21 Jan 2021 14:52:44 +0000"  >&lt;p&gt;I have not seen the issue again since downgrading the FCV to 4.2 (still same binaries).&lt;/p&gt;</comment>
                            <comment id="3573280" author="dmitry.agranat" created="Wed, 20 Jan 2021 13:07:28 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;, did you have a chance to collect a core dump?&lt;/p&gt;

&lt;p&gt;As an update, this issue is now being investigated in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52735&quot; title=&quot;mongodb crash with &amp;quot;Invariant failure&amp;quot; opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52735&quot;&gt;&lt;del&gt;SERVER-52735&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3553785" author="jonathan.streets" created="Thu, 7 Jan 2021 21:55:27 +0000"  >&lt;p&gt;hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;, I was able to test these instructions to produce a coredump on amazon linux 2. Please let us know how it goes.&#160; Also, please note that a core dump contains PII and will consume approximately as much disk space as the virtual memory in use by the process when it crashes.&#160;jon&lt;/p&gt;

&lt;p&gt;Edit &lt;tt&gt;/etc/systemd/system.conf&lt;/tt&gt; and add&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;tt&gt;DumpCore=yes&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;DefaultLimitCORE=infinity&lt;/tt&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Restart daemon with&lt;br/&gt;
 &lt;tt&gt;systemctl daemon-reexec&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;Change the default coredump name to pipe it to the coredump utility (all on one line):&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;echo &apos;|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e&apos; &amp;gt; /proc/sys/kernel/core_pattern&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Make the new core name persist over reboots by changing &lt;tt&gt;/etc/sysctl.conf&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;Reload the new configuration with&#160;&lt;br/&gt;
 &lt;tt&gt;sysctl -p&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;Check the new configuration with &lt;tt&gt;sysctl -a | grep kernel | grep core&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;Set the core file size from 0 to unlimited:&lt;br/&gt;
 &lt;tt&gt;ulimit -c unlimited&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Check with&lt;br/&gt;
 &lt;tt&gt;ulimit -a&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Restart MongoDB (or reboot)&lt;br/&gt;
 &lt;tt&gt;systemctl restart mongod&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Test by sending &lt;tt&gt;SIGSEGV&lt;/tt&gt; to &lt;tt&gt;mongod&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;ps -ef | grep mongod&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;kill -11 NNNNN&lt;/tt&gt; (where NNNNN is the PID of mongod)&lt;/p&gt;

&lt;p&gt;The coredump should show up in&#160;&lt;br/&gt;
 &lt;tt&gt;systemd-coredumpctl list&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;And in&lt;br/&gt;
 &lt;tt&gt;ls /var/lib/systemd/coredump&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="3551134" author="b.granetzke@fetchrewards.com" created="Wed, 6 Jan 2021 18:53:07 +0000"  >&lt;p&gt;Also, just an FYI:&#160; I&apos;m still running 4.4.2-ent binaries, but I recently set FCV back to 4.2 in prep for a downgrade.&#160; I&apos;ve convinced management to hold off on the downgrade until one more primary failure (e.g. the pause).&#160; I will grab the diagnostic info when that occurs and whatever else you need.&#160; I assume the FCV isn&apos;t related to the underlying cause, but I thought I&apos;d mention it in case it&apos;s relevant.&lt;/p&gt;</comment>
                            <comment id="3551097" author="b.granetzke@fetchrewards.com" created="Wed, 6 Jan 2021 18:40:20 +0000"  >&lt;p&gt;latest release of amazon linux 2...appreciate the help&lt;/p&gt;</comment>
                            <comment id="3550994" author="jonathan.streets" created="Wed, 6 Jan 2021 18:01:23 +0000"  >&lt;p&gt;hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;core dumps aren&apos;t on by default, and the commands to turn on and collect them depend very much on the operating system.&#160; i see that its an Amazon 2. i&apos;ll look for some commands.&lt;/p&gt;

&lt;p&gt;&#160;jon&#160;&lt;/p&gt;</comment>
                            <comment id="3550928" author="b.granetzke@fetchrewards.com" created="Wed, 6 Jan 2021 17:37:20 +0000"  >&lt;p&gt;@Dmitry I tried finding them with the prior crashes but could not find any.&#160; Do invariant failures produce them?&#160; if so, where should I look?&lt;/p&gt;</comment>
                            <comment id="3550162" author="dmitry.agranat" created="Wed, 6 Jan 2021 09:47:14 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;, do you happen to have a core dump from one of the previous events? If not, would it be possible to save one if this failure reoccurs?&lt;/p&gt;</comment>
                            <comment id="3546310" author="dmitry.agranat" created="Mon, 4 Jan 2021 11:19:31 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt; for uploading the data, we are looking into this and will update you on our findings. Just to confirm, do you still have &lt;tt&gt;diagnostic.data&lt;/tt&gt; for either of the reported events? If not, as Bruce mentioned above, if this happens again, it would be helpful to collect logs and &lt;tt&gt;diagnostic.data&lt;/tt&gt;; that might have more information about the freeze.&lt;/p&gt;</comment>
                            <comment id="3542855" author="b.granetzke@fetchrewards.com" created="Mon, 28 Dec 2020 16:48:59 +0000"  >&lt;p&gt;@Bruce: I can send you logs from all three servers for additional freeze events if you would like data for more than the attached two.&#160; The pattern is very similar for all of them.&#160; Basically one of the secondaries gets a socket timeout for the primary and initiates an election.&#160; My specific search string to quantify the situation is:&#160;msg=&quot;Starting an election, since we&apos;ve seen no PRIMARY in election timeout period&quot;.&#160; I can take the timestamp of the prior NetworkTimeout on the secondary and look on the primary and can see a distinct gap in log events at that time (give or take milliseconds).&lt;/p&gt;</comment>
                            <comment id="3542726" author="b.granetzke@fetchrewards.com" created="Mon, 28 Dec 2020 14:59:22 +0000"  >&lt;p&gt;I&apos;ve uploaded the logs for the two invariant failure events.&#160; The file names indicate the exact log time of the oldest invariant failure for the event.&#160; In both cases, all three nodes ended up crashing with the same invariant failure.&#160; The first event was on v4.4.1 and the second was on 4.4.2-ent.&lt;/p&gt;

&lt;p&gt;Regarding the &quot;pause&quot;, both events started with the primary pausing long enough for it to miss the heartbeat.&#160; For the 12/19 event, the pause was on the mongo2 server between 2020-12-19T21:21:02.971 and 2020-12-19T21:21:11.893.&#160; You&apos;ll notice based on log rate prior to the pause, a 9 second period with no logs is extremely unlikely.&lt;/p&gt;

&lt;p&gt;The pause events are much more frequent than the invariant failures.&#160; We&apos;ve had 18 random primary failures since the upgrade to 4.4 (8-10 weeks ago).&#160; The most recent was 12/23, but the diagnostic data has already rolled since that event as well.&lt;/p&gt;</comment>
                            <comment id="3540850" author="bruce.lucas@10gen.com" created="Wed, 23 Dec 2020 14:50:26 +0000"  >&lt;p&gt;Also the freeze that you mention is interesting. If you are willing it would be helpful if you could hold off on reverting to 4.2 until it happens one more time, then collect logs and diagnostic.data; that might have more information about the freeze.&lt;/p&gt;</comment>
                            <comment id="3540844" author="bruce.lucas@10gen.com" created="Wed, 23 Dec 2020 14:47:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt; this comment you posted on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52735&quot; title=&quot;mongodb crash with &amp;quot;Invariant failure&amp;quot; opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52735&quot;&gt;&lt;del&gt;SERVER-52735&lt;/del&gt;&lt;/a&gt; seems especially interesting:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Also, twice, it crashed all three nodes on the cluster within seconds.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I think as you suggested logs in a 2 hour window would be useful, especially if you have logs for all 3 nodes that crashed simultaneously.&lt;/p&gt;</comment>
                            <comment id="3540793" author="b.granetzke@fetchrewards.com" created="Wed, 23 Dec 2020 14:15:23 +0000"  >&lt;p&gt;Unfortunately the metrics files have rolled since the most recent event (on the 19th).&#160; The log files have rolled as well, but I can pull them from cloudwatch logs.&#160; It is a lot of data.&#160; Should I just send you 1 hour windows before/after each of the 3 events?&lt;/p&gt;</comment>
                            <comment id="3540642" author="dmitry.agranat" created="Wed, 23 Dec 2020 10:27:11 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=b.granetzke%40fetchrewards.com&quot; class=&quot;user-hover&quot; rel=&quot;b.granetzke@fetchrewards.com&quot;&gt;b.granetzke@fetchrewards.com&lt;/a&gt;, thank you for the report.&lt;/p&gt;

&lt;p&gt;Would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/a1afa8da-a745-4424-818e-88da78048993.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3540150" author="b.granetzke@fetchrewards.com" created="Tue, 22 Dec 2020 20:31:32 +0000"  >&lt;p&gt;Actually, my issues looks a lot like&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52735&quot; title=&quot;mongodb crash with &amp;quot;Invariant failure&amp;quot; opCtx != nullptr &amp;amp;&amp;amp; _opCtx == nullptr&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-52735&quot;&gt;&lt;del&gt;SERVER-52735&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1537998">SERVER-52735</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1537998">SERVER-52735</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1580342">SERVER-53566</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="293144" name="2020-11-09T00_16_19.496.zip" size="39071583" author="b.granetzke@fetchrewards.com" created="Mon, 28 Dec 2020 14:30:11 +0000"/>
                            <attachment id="293152" name="2020-12-19T21_21_13.189.zip" size="38971687" author="b.granetzke@fetchrewards.com" created="Mon, 28 Dec 2020 14:40:51 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>17.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 23 Dec 2020 10:27:11 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 2 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 2 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>b.granetzke@fetchrewards.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>jonathan.streets@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyn6gn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hy8zo7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hymspz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>