<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:59:11 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-63984] Primary replica member becomes unavailable during normal operation</title>
                <link>https://jira.mongodb.org/browse/SERVER-63984</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;For no apparent reason, our primary replica member of one of the shards got unresponsive until we restarted it.&lt;/p&gt;

&lt;p&gt;The incident lasted for about 35 minutes. During that time we saw almost 100% consumption on the primary and its load average was up to 60 times the normal values.&lt;/p&gt;

&lt;p&gt;From the logs (as of the beginning of the incident) we understood only this:&lt;br/&gt;
1) The amount of opened connections started increasing.&lt;br/&gt;
2) Some opened cursors got timed out.&lt;br/&gt;
3) The pooled connections to other members got dropped (due to shutdown, but we didn&apos;t try to shut down the primary at that time).&lt;br/&gt;
```&lt;br/&gt;
I  CONNPOOL &lt;span class=&quot;error&quot;&gt;&amp;#91;TaskExecutorPool-0&amp;#93;&lt;/span&gt; Dropping all pooled connections to some-secondary:27017 due to ShutdownInProgress: Pool for some-secondary:27017 has expired.&lt;br/&gt;
```&lt;br/&gt;
4) After some time no log entries appear (for about 25 minutes) until we restarted the primary.&lt;/p&gt;

&lt;p&gt;Our cluster configuration:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;shard cluster with 10 shards&lt;/li&gt;
	&lt;li&gt;four replicas in each shard&lt;/li&gt;
	&lt;li&gt;about 400 GB of data in storage size per shard&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Replica server configuration:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Ubuntu 16.04&lt;/li&gt;
	&lt;li&gt;XSF&lt;/li&gt;
	&lt;li&gt;Kernel - 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux&lt;/li&gt;
	&lt;li&gt;Disable Transparent Huge disabled&lt;/li&gt;
	&lt;li&gt;AWS m5.large (2cpu\8gb)&lt;/li&gt;
	&lt;li&gt;SSD GP3 450 Gb&lt;/li&gt;
	&lt;li&gt;monogo-org-server - 4.2.17&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;`diagnostic.data` of the primary and one of the secondary attached to the post.&lt;/p&gt;</description>
                <environment>Ubuntu 16.04&lt;br/&gt;
XSF&lt;br/&gt;
Kernel - 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux&lt;br/&gt;
Disable Transparent Huge disabled&lt;br/&gt;
AWS m5.large (2cpu\8gb)&lt;br/&gt;
SSD GP3 450 Gb&lt;br/&gt;
monogo-org-server - 4.2.17</environment>
        <key id="1990131">SERVER-63984</key>
            <summary>Primary replica member becomes unavailable during normal operation</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13204">Community Answered</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="vladimirred456@gmail.com">Vladimir Beliakov</reporter>
                        <labels>
                    </labels>
                <created>Fri, 25 Feb 2022 08:39:07 +0000</created>
                <updated>Fri, 27 Oct 2023 15:56:25 +0000</updated>
                            <resolved>Thu, 23 Jun 2022 10:03:23 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4678746" author="vladimirred456@gmail.com" created="Thu, 14 Jul 2022 10:10:48 +0000"  >&lt;p&gt;I suspect this has something to do with balancing chunks. We&apos;re using `sh.addTagRange` to set custom chunk ranges. Yesterday we changed the ranges and the balancer started moving chunks. The incident happened 15-20 minutes afterwards.&#160;&lt;/p&gt;</comment>
                            <comment id="4676502" author="vladimirred456@gmail.com" created="Wed, 13 Jul 2022 13:58:25 +0000"  >&lt;p&gt;I found this message in the logs&lt;/p&gt;


&lt;blockquote&gt;&lt;p&gt;&lt;tt&gt;STORAGE &#160;&lt;span class=&quot;error&quot;&gt;&amp;#91;FlowControlRefresher&amp;#93;&lt;/span&gt; Flow control is engaged and the sustainer point is not moving. Please check the health of all secondaries.&lt;/tt&gt;&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;which led me to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53738&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this issue&lt;/a&gt;. Might be related.&lt;/p&gt;</comment>
                            <comment id="4676486" author="vladimirred456@gmail.com" created="Wed, 13 Jul 2022 13:52:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat@mongodb.com&quot;&gt;dmitry.agranat@mongodb.com&lt;/a&gt;&lt;br/&gt;
We ran into a somewhat similar issue but the behavior of the primary was a bit different. The transactions may not be to blame. So, I&apos;m kindly asking to analyze the diagnostic data to help us understand what was the culprit.&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/362632/362632_diagnostic.data.zip&quot; title=&quot;diagnostic.data.zip attached to SERVER-63984&quot;&gt;diagnostic.data.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="4603880" author="dmitry.agranat" created="Thu, 9 Jun 2022 07:30:59 +0000"  >&lt;p&gt;Thank you &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;, after looking at the latest stack traces from May 12th, this is indeed what we&apos;ve suspected from the start: when a dirty portion of the WT data gets to 20%, we are unable to clean any of this dirty data and, under certain conditions, transactions can get stuck.&lt;/p&gt;

&lt;p&gt;There are a few steps you can take to address this issue:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Most effective way to avoid this issue is to diagnose why a transaction pinning so much dirty data and try to fix this on the application side. In other words, make sure your transactions (that change data) are short and do not need to &quot;dirty&quot; more that the 20% of the WT cache (in your, case, more than ~600MB)&lt;/li&gt;
	&lt;li&gt;Increase your memory to an extent where your transactions (that change data) do not reach filling out 20% of the WT cache in a single transaction&lt;/li&gt;
	&lt;li&gt;Upgrade to MongoDB 5.0 where some significant improvements were made to improve this area&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="4592256" author="vladimirred456@gmail.com" created="Fri, 3 Jun 2022 07:15:19 +0000"  >&lt;p&gt;Hi&#160;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat@mongodb.com&quot;&gt;dmitry.agranat@mongodb.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We one more time had approximately the same issue. However, this time the machine became unresponsive, so we couldn&apos;t collect the stack traces, only diagnostic data (attached).&lt;br/&gt;
The incident started at about 06/03/2022 05:29:00 UTC.&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/362632/362632_diagnostic.data.zip&quot; title=&quot;diagnostic.data.zip attached to SERVER-63984&quot;&gt;diagnostic.data.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="4545126" author="vladimirred456@gmail.com" created="Thu, 12 May 2022 14:58:40 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat@mongodb.com&quot;&gt;dmitry.agranat@mongodb.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We ran into the same issue and were able to collect the stack traces during the incident. You can find them and the diagnostic.data attached to this message.&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/378711/378711_data.diagnostics.zip&quot; title=&quot;data.diagnostics.zip attached to SERVER-63984&quot;&gt;data.diagnostics.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/378712/378712_gdb_2022-05-12_09-59-24.zip&quot; title=&quot;gdb_2022-05-12_09-59-24.zip attached to SERVER-63984&quot;&gt;gdb_2022-05-12_09-59-24.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="4487664" author="vladimirred456@gmail.com" created="Mon, 18 Apr 2022 09:44:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat@mongodb.com&quot;&gt;dmitry.agranat@mongodb.com&lt;/a&gt; thanks for your answer. I&apos;ll keep that in mind.&lt;/p&gt;</comment>
                            <comment id="4487650" author="dmitry.agranat" created="Mon, 18 Apr 2022 09:09:27 +0000"  >&lt;p&gt;Thanks for the clarification &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;. Since the stacks were collected only &lt;b&gt;after&lt;/b&gt; the node was stepped down, we no longer see the reason why the previous member state was unavailable. In order for us to get to the bottom of this issue, we need stack traces collected on the relevant node &lt;b&gt;while&lt;/b&gt; the issue is still in progress.&lt;/p&gt;</comment>
                            <comment id="4487615" author="vladimirred456@gmail.com" created="Mon, 18 Apr 2022 08:34:39 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat@mongodb.com&quot;&gt;dmitry.agranat@mongodb.com&lt;/a&gt;, we had to make that member step down during the incident and we collected the stack traces right after that. Here&apos;s diagnostic.data&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/373844/373844_metrics.zip&quot; title=&quot;metrics.zip attached to SERVER-63984&quot;&gt;metrics.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
glibc version:&lt;br/&gt;
ldd (Ubuntu GLIBC 2.23-0ubuntu11.3) 2.23&lt;br/&gt;
Copyright (C) 2016 Free Software Foundation, Inc.&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                            <comment id="4487085" author="dmitry.agranat" created="Sun, 17 Apr 2022 10:38:28 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;, do you also have &lt;tt&gt;diagnostic.data&lt;/tt&gt; covering the time of the latest event that happened on April 12th? I am asking this because based on the stack traces provided, the state of the member does not look to be Primary. Can you please confirm this?&lt;/p&gt;

&lt;p&gt;In addition, could you please post your current &lt;tt&gt;glibc&lt;/tt&gt; version?&lt;/p&gt;</comment>
                            <comment id="4476994" author="vladimirred456@gmail.com" created="Tue, 12 Apr 2022 14:29:47 +0000"  >&lt;p&gt;Hi, Edwin Zhou!&lt;/p&gt;

&lt;p&gt;We had a somewhat similar problem. The required stack traces file is attached. I hope that will be helpful.&lt;/p&gt;

&lt;p&gt;Cheers!&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/372671/372671_gdb_2022-04-12_14-15-43.txt&quot; title=&quot;gdb_2022-04-12_14-15-43.txt attached to SERVER-63984&quot;&gt;gdb_2022-04-12_14-15-43.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="4410417" author="JIRAUSER1257066" created="Mon, 14 Mar 2022 16:19:56 +0000"  >&lt;p&gt;Thank you for your follow up, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;. I will leave this on waiting for user input as we await the stack traces from a repeat occurrence.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4405889" author="vladimirred456@gmail.com" created="Fri, 11 Mar 2022 07:06:34 +0000"  >&lt;p&gt;Hi,&#160;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=edwin.zhou&quot; class=&quot;user-hover&quot; rel=&quot;edwin.zhou&quot;&gt;edwin.zhou&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;Yes, I already made our devops team aware of collecting the stack traces in case we have the same problem. But no similar incidents have happened yet. When we have the stack traces I&apos;ll attach them to the ticker ASAP.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thank you for your help!&lt;/p&gt;</comment>
                            <comment id="4404853" author="JIRAUSER1257066" created="Thu, 10 Mar 2022 20:12:03 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We still need additional information to diagnose the problem. If this is still an issue for you, would you please collect the gdb stack traces and upload them to this ticket?&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4379657" author="dmitry.agranat" created="Mon, 28 Feb 2022 08:31:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vladimirred456%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vladimirred456@gmail.com&quot;&gt;vladimirred456@gmail.com&lt;/a&gt;, after looking at the data you&apos;ve practically provided (thank you for that), I suspect this might be related to a particular situation where a process can get stuck with long-running transactions not aborting. This usually happens when a dirty portion of the WT data gets to 20%, we are unable to clean any of this dirty data and, under certain conditions, transactions can get stuck.&lt;/p&gt;

&lt;p&gt;In order to confirm or rule this out, we&apos;ll need to collect some additional information if this issue occurs again. After the process gets stuck and before rebooting the process, please execute this command:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gdb -p $(pidof mongod) -batch -ex &apos;thread apply all bt&apos; &amp;gt;gdb_`date +&quot;%Y-%m-%d_%H-%M-%S&quot;`.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will collect stack traces showing where threads are being stuck. Please test &lt;tt&gt;gdb&lt;/tt&gt; command to make sure it works as expected if/when the issue occurs again.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="378711" name="data.diagnostics.zip" size="15367725" author="vladimirred456@gmail.com" created="Thu, 12 May 2022 14:58:36 +0000"/>
                            <attachment id="382869" name="diagnostic.data-1.zip" size="14785649" author="vladimirred456@gmail.com" created="Fri, 3 Jun 2022 07:15:19 +0000"/>
                            <attachment id="389938" name="diagnostic.data-2.zip" size="20812235" author="vladimirred456@gmail.com" created="Wed, 13 Jul 2022 13:51:28 +0000"/>
                            <attachment id="362632" name="diagnostic.data.zip" size="50721967" author="vladimirred456@gmail.com" created="Fri, 25 Feb 2022 08:34:00 +0000"/>
                            <attachment id="373399" name="gdb.html" size="27506" author="edwin.zhou@mongodb.com" created="Thu, 14 Apr 2022 15:50:18 +0000"/>
                            <attachment id="372671" name="gdb_2022-04-12_14-15-43.txt" size="548119" author="vladimirred456@gmail.com" created="Tue, 12 Apr 2022 14:29:29 +0000"/>
                            <attachment id="378712" name="gdb_2022-05-12_09-59-24.zip" size="149762" author="vladimirred456@gmail.com" created="Thu, 12 May 2022 14:58:40 +0000"/>
                            <attachment id="373844" name="metrics.zip" size="33232377" author="vladimirred456@gmail.com" created="Mon, 18 Apr 2022 08:31:19 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 28 Feb 2022 08:31:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 29 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 29 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>vladimirred456@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0lbl3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i04dkk:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0kxqf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>