<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26867] Timeout-related assert.soon failures should trigger the hang analyzer</title>
                <link>https://jira.mongodb.org/browse/SERVER-26867</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When an test fails due to a timeout, it would be nice if that would trigger the hang analyzer before tearing down the processes. This would let us see what the server is doing that is preventing it from doing what we expect.&lt;/p&gt;

&lt;p&gt;&lt;del&gt;Timeout-related failures include:&lt;/del&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;del&gt;assert.soon()&lt;/del&gt;&lt;/li&gt;
	&lt;li&gt;&lt;del&gt;wtimeout expiring&lt;/del&gt;&lt;/li&gt;
	&lt;li&gt;&lt;del&gt;$maxTimeMs expiring&lt;/del&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Update: Per discussion below we will be limiting this to &lt;tt&gt;assert.soon&lt;/tt&gt;, &lt;tt&gt;assert.soonNoExcept&lt;/tt&gt;, &lt;tt&gt;assert.retry&lt;/tt&gt;, and &lt;tt&gt;assert.assert.retryNoExcept&lt;/tt&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="328471">SERVER-26867</key>
            <summary>Timeout-related assert.soon failures should trigger the hang analyzer</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ryan.timmons@mongodb.com">Ryan Timmons</assignee>
                                    <reporter username="mathias@mongodb.com">Mathias Stearn</reporter>
                        <labels>
                            <label>stm</label>
                            <label>tig-hanganalyzer</label>
                    </labels>
                <created>Tue, 1 Nov 2016 17:21:24 +0000</created>
                <updated>Tue, 19 Nov 2019 19:56:50 +0000</updated>
                            <resolved>Thu, 31 Oct 2019 21:09:59 +0000</resolved>
                                                    <fixVersion>4.3.1</fixVersion>
                                    <component>Testing Infrastructure</component>
                                        <votes>11</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="2513376" author="ryan.timmons" created="Thu, 31 Oct 2019 21:09:59 +0000"  >&lt;p&gt;This has been released. assert.soon, assert.retry, and assert.time will now call the hang-analyzer prior to throwing.&lt;/p&gt;</comment>
                            <comment id="2513368" author="xgen-internal-githook" created="Thu, 31 Oct 2019 21:07:08 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ryan Timmons&apos;, &apos;username&apos;: &apos;rtimmons&apos;, &apos;email&apos;: &apos;ryan.timmons@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26867&quot; title=&quot;Timeout-related assert.soon failures should trigger the hang analyzer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26867&quot;&gt;&lt;del&gt;SERVER-26867&lt;/del&gt;&lt;/a&gt; Call hang_analyzer.py from assert functions&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f97d73eefc1a4302e949d9cbc85d89fd07847ee6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f97d73eefc1a4302e949d9cbc85d89fd07847ee6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2485467" author="judah.schvimer" created="Wed, 16 Oct 2019 18:51:30 +0000"  >&lt;p&gt;SGTM. wtimeout and maxTimeMS timeouts we&apos;ve decided to largely stop providing so that tests hang until the process kills them and provides the hang analyzer output. This may not be ideal, but it works.&lt;/p&gt;</comment>
                            <comment id="2485433" author="ryan.timmons" created="Wed, 16 Oct 2019 18:37:34 +0000"  >&lt;p&gt;You now call &lt;tt&gt;MongoRunner.runHangAnalyzer()&lt;/tt&gt; in the shell to invoke &lt;tt&gt;hang_analyzer.py&lt;/tt&gt; on all processes started by the shell and resmoke. This was done in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43254&quot; title=&quot;Hang Analyzer shell integration uses child and peer mongo processes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43254&quot;&gt;&lt;del&gt;SERVER-43254&lt;/del&gt;&lt;/a&gt;. (There&apos;s some followup work to get it working when debug symbols haven&apos;t already been inflated - &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44017&quot; title=&quot;Hang Analyzer Unzips Debug Symbols&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44017&quot;&gt;&lt;del&gt;SERVER-44017&lt;/del&gt;&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;It is trivial to call this from &lt;tt&gt;assert.soon&lt;/tt&gt; prior to actually throwing. But the original text of this ticket indicates also calling this for &lt;tt&gt;wtimeout&lt;/tt&gt; and &lt;tt&gt;$maxTimeMs&lt;/tt&gt; expiring.&lt;/p&gt;

&lt;p&gt;I&apos;m less confident with how we&apos;d actually hook into those mechamisms within the shell without doing some trickery to monkeypatch calls to &lt;tt&gt;runCommand&lt;/tt&gt; and inspect args for &lt;tt&gt;wtimeout&lt;/tt&gt;/&lt;tt&gt;$maxTimeMs&lt;/tt&gt; (same argument applies to &lt;tt&gt;assert.commandWorked&lt;/tt&gt;).&lt;/p&gt;

&lt;p&gt;It seems like we get most of the benefit from&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Having &lt;tt&gt;MongoRunner.runHangAnalyzer()&lt;/tt&gt; as a thing that can be dropped in as-needed into &lt;tt&gt;catch&lt;/tt&gt; blocks, and&lt;/li&gt;
	&lt;li&gt;Automatically calling this from &lt;tt&gt;assert.soon&lt;/tt&gt;, &lt;tt&gt;assert.soonNoExcept&lt;/tt&gt;, &lt;tt&gt;assert.retry&lt;/tt&gt;, and &lt;tt&gt;assert.assert.retryNoExcept&lt;/tt&gt;.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Unless there&apos;s a simple way to incorporate this function-call to other timeout scenarios, I propose that we don&apos;t automatically run &lt;tt&gt;hang_analyzer.py&lt;/tt&gt; for anything other than these 4 &lt;tt&gt;assert&lt;/tt&gt; methods. This aligns with the original scope-doc for &lt;tt&gt;hang_analyzer.py&lt;/tt&gt; integration PM-1546 but I&apos;m curious if there is a pressing need for more and/or an easy way to get these other things.&lt;/p&gt;

&lt;p&gt;Open questions to anyone watching this ticket:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Is there a straightforward way to inject calls to &lt;tt&gt;hang_analyzer.py&lt;/tt&gt; in all places where a query/command may have &lt;tt&gt;wtimeout&lt;/tt&gt;; similar for &lt;tt&gt;maxTimeMs&lt;/tt&gt;?&lt;/li&gt;
	&lt;li&gt;How actually usuful is automatically calling &lt;tt&gt;hang_analyzer.py&lt;/tt&gt; from places other than &lt;tt&gt;assert.soon&lt;/tt&gt; given you can detect timeouts with exceptions and call &lt;tt&gt;MongoRunner.runHangAnalyzer()&lt;/tt&gt; within test-logic?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;If no objections in the coming day or two I&apos;ll update this ticket text/description to reflect just updates to &lt;tt&gt;assert.soon&lt;/tt&gt; and friends. We can evaluate incorporating it further in future tickets.&lt;/p&gt;

&lt;p&gt;Please feel free to reach out to me/STM on slack if easier.&lt;/p&gt;</comment>
                            <comment id="2361734" author="max.hirschhorn@10gen.com" created="Wed, 7 Aug 2019 18:36:44 +0000"  >&lt;blockquote&gt;
&lt;p&gt;Max Hirschhorn and Robert Guo, what is the timeline for doing #2?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;We&apos;re planning to write a scope document for this work in the next iteration. I don&apos;t imagine it&apos;ll need a design document but should have a sketch of the implementation as an appendix for reviewers to read through. It&apos;d be worth having a conversation as to whether we&apos;d want to bubble this up over the work on the query and aggregation fuzzers currently in-progress and the performance testing with slow networks that was slated to be next.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Would a repl team member be able to do it effectively?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Let&apos;s decide on that after the scope document (esp. the appendix section) is written.&lt;/p&gt;</comment>
                            <comment id="2361695" author="judah.schvimer" created="Wed, 7 Aug 2019 18:21:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn&quot;&gt;max.hirschhorn&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=robert.guo&quot; class=&quot;user-hover&quot; rel=&quot;robert.guo&quot;&gt;robert.guo&lt;/a&gt;, what is the timeline for doing #2? Would a repl team member be able to do it effectively?&lt;/p&gt;</comment>
                            <comment id="2360440" author="max.hirschhorn@10gen.com" created="Tue, 6 Aug 2019 21:29:56 +0000"  >&lt;p&gt;I chatted with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; in-person yesterday. I believe he and I are on the same page about the local testing experience and why we&apos;d rather consider having different timeouts of &quot;forever&quot; when running in Evergreen versus when running locally. (I stand by 5 minutes is already too long to have to wait locally to find out you wrote the wrong condition inside of an &lt;tt&gt;assert.soon()&lt;/tt&gt; while iterating on writing/debuggin a test, but it&apos;d be challenging to reverse &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25977&quot; title=&quot;Increase default assert.soon timeout to 5 minutes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25977&quot;&gt;&lt;del&gt;SERVER-25977&lt;/del&gt;&lt;/a&gt; at this point.)&lt;/p&gt;

&lt;p&gt;To summarize the different options for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26867&quot; title=&quot;Timeout-related assert.soon failures should trigger the hang analyzer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26867&quot;&gt;&lt;del&gt;SERVER-26867&lt;/del&gt;&lt;/a&gt;:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;We could increase the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f69bdec6b5e6a8d0e3162082dce5035e40bc2269/src/mongo/shell/assert.js#L326&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;assert.soon()&lt;/tt&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f69bdec6b5e6a8d0e3162082dce5035e40bc2269/src/mongo/shell/replsettest.js#L2778&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;ReplSetTest.kDefaultTimeoutMS&lt;/tt&gt;&lt;/a&gt; timeout to 24 hours when running in Evergreen. (We would leave their current values when running locally.) The complication with this approach is that tests which run for a long period of time produce an obnoxious amount of log output, which makes the Build Baron tooling owned by the DAG team unhappy. While the Evergreen task would eventually time out, it may be after 30 minutes, 2 hours, or the &lt;tt&gt;timeout.update&lt;/tt&gt; value that&apos;s been set.&lt;/li&gt;
	&lt;li&gt;We could run &lt;tt&gt;python buildscripts/hang_analyzer.py ...&lt;/tt&gt; from within &lt;tt&gt;assert.soon()&lt;/tt&gt; when it would throw a JavaScript exception. This is more involved than the approach in #1 because we&apos;d need to &amp;#40;i) ask the &lt;tt&gt;ProgramRunner&lt;/tt&gt; instance in the mongo shell which MongoDB processes it spawned, and (ii) plumb the pids of the MongoDB processes spawned by the resmoke.py job thread for the currently running test.
	&lt;ul&gt;
		&lt;li&gt;One question may be whether there should be a JavaScript function which can be called from a test directly that is a no-op when run locally and runs &lt;tt&gt;python buildscripts/hang_analyzer.py ...&lt;/tt&gt; in Evergreen. This would allow the debugger to be attached at an arbitrary point in the test and not just during &lt;tt&gt;assert.soon()&lt;/tt&gt;.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I had imagined we&apos;d achieve this ticket by doing #2.&lt;/p&gt;</comment>
                            <comment id="2358204" author="william.schultz" created="Mon, 5 Aug 2019 18:13:01 +0000"  >&lt;p&gt;BF-14272 (investigated by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=samy.lanka&quot; class=&quot;user-hover&quot; rel=&quot;samy.lanka&quot;&gt;samy.lanka&lt;/a&gt;) is another case where getting stack traces after an awaitReplication timeout would have been very helpful. In a recent replication meeting we raised the question of why it was not considered acceptable to simply raise the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47d0308cd596c6d40d6a3069379be1bdaf51d47b/src/mongo/shell/replsettest.js#L1371&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;default awaitReplication timeout&lt;/a&gt; to match the task timeout, so that the timeouts would trigger the hang analyzer. We thought that there may have been a prior objection to this on the grounds that it would make the local testing experience worse, but the details of the objection were not entirely clear. It would be good to clarify what the possible objections may be.&lt;/p&gt;

&lt;p&gt;Another proposal that came up was to allow the awaitReplication timeout to be configured separately when running in Evergreen versus running locally. This would avoid any negative side effects on the local testing experience. One option would be to allow the awaitReplication timeout to be overriden by setting a &quot;TestData&quot; parameter. Something similar was done for altering log verbosity as a part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32907&quot; title=&quot;Turn heartbeat log verbosity back down in tests&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32907&quot;&gt;&lt;del&gt;SERVER-32907&lt;/del&gt;&lt;/a&gt;. We could, for example, have an &quot;awaitReplicationTimeout&quot; parameter that we could set to the appropriate task timeout when running in Evergreen, but it could default to the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47d0308cd596c6d40d6a3069379be1bdaf51d47b/src/mongo/shell/replsettest.js#L2783&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;existing value&lt;/a&gt; (10 minutes), when tests are running locally. This would not necessarily fix the issue for all varieties of these timeouts (e.g. wtimeout, assert.soon), but starting with awaitReplication could be a good step.&lt;/p&gt;</comment>
                            <comment id="2293799" author="siyuan.zhou@10gen.com" created="Fri, 21 Jun 2019 23:23:13 +0000"  >&lt;p&gt;BF-13678 doesn&apos;t have sufficient evidence to know what&apos;s causing the timeout. Core dump would also be very helpful when awaitReplication() times out.&lt;/p&gt;</comment>
                            <comment id="1555411" author="dianna.hohensee" created="Fri, 21 Apr 2017 23:03:53 +0000"  >&lt;p&gt;To jump on the bandwagon of providing use-cases incentives for this improvement:&lt;/p&gt;

&lt;p&gt;Would be nice for assert.soon failures like BF-5335, where we&apos;re concerned about deadlock because nothing is happening inexplicably and nothing of interest is getting logged (and we&apos;ve been doing significant sharding code additions). I&apos;m going to try to artificially provoke the hang analyzer, somehow, to get stack dumps along with bumping the verbosity in hopes of gleaning more information &amp;#8211; this ticket would do it all for me, apparently &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1433531" author="judah.schvimer" created="Mon, 14 Nov 2016 18:36:59 +0000"  >&lt;p&gt;This would make BF-3913 much easier to debug. It appears to be a hang elsewhere in the system that leads to writeConcern timing out. Knowing what else is happening on the secondary would make it clear what was hanging and causing the timeout.&lt;/p&gt;</comment>
                            <comment id="1433528" author="redbeard0531" created="Mon, 14 Nov 2016 18:35:20 +0000"  >&lt;p&gt;Expanding scope to include other timeouts after seeing a hard to debug failure of wtimeout.&lt;/p&gt;</comment>
                            <comment id="1423151" author="milkie" created="Tue, 1 Nov 2016 17:25:55 +0000"  >&lt;p&gt;Or rather, the threads are working normally (or are all idle), and it is the logic of the test that set up the failure to begin with.&lt;/p&gt;</comment>
                            <comment id="1423150" author="judah.schvimer" created="Tue, 1 Nov 2016 17:25:53 +0000"  >&lt;p&gt;If we do this, assert.retry() should also probably be included.&lt;/p&gt;</comment>
                            <comment id="1423147" author="milkie" created="Tue, 1 Nov 2016 17:25:13 +0000"  >&lt;p&gt;I don&apos;t think this would be that useful.  The overwhelming majority of assert.soon failures that I&apos;ve seen are failing without the server threads actually being hung.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="922746">SERVER-43256</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="922745">SERVER-43255</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="211594">SERVER-19030</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="746224">SERVER-40857</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="305141">SERVER-25346</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1012081">SERVER-44739</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 1 Nov 2016 17:25:13 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 14 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-43256'>SERVER-43256</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1546</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 14 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>ryan.timmons@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjs3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsrfqv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3318">STM 2019-10-31</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsee4f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>