<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:38:11 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-56054] Change minThreads value for replication writer thread pool to 0</title>
                <link>https://jira.mongodb.org/browse/SERVER-56054</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54805&quot; title=&quot;Mongo become unresponsive, Spike in Connections and FD&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54805&quot;&gt;&lt;del&gt;SERVER-54805&lt;/del&gt;&lt;/a&gt; describes a case where the replication machinery on replica set secondaries ceases to make progress, with the symptom being that all threads in the replication writer thread pool are idle and the thread driving secondary replication is simultaneously blocked waiting for those writer threads to finish their work.&lt;/p&gt;

&lt;p&gt;So far, this behavior has only manifest on systems with glibc versions susceptible to &lt;a href=&quot;https://sourceware.org/bugzilla/show_bug.cgi?id=25847&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this glibc pthread condition variable bug&lt;/a&gt;. While I have not been able to build a minimal reproduction using our &lt;tt&gt;ThreadPool&lt;/tt&gt; type, the scenario proven to exist in &lt;a href=&quot;https://probablydance.com/2020/10/31/using-tla-in-the-real-world-to-understand-a-glibc-bug/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this blog post about using TLA+ to model glibc condition variables&lt;/a&gt; is perfectly analogous to how replication uses thread pools. In this scenario, a signal delivery that is lost due to the glibc bug leads to incomplete work being left in the thread pool, and no threads waking up to perform the work.&lt;/p&gt;

&lt;p&gt;Fortunately, a low-risk workaround for this bug as it manifests in the replication system&apos;s use of ThreadPool exists. By setting &lt;tt&gt;minThreads&lt;/tt&gt; to 0 instead of its current value, which is equal to &lt;tt&gt;maxThreads&lt;/tt&gt;, we ensure that all waits performed by worker threads eventually wake up due to expiration of the idle thread timeout.&lt;/p&gt;

&lt;p&gt;The task in this ticket is to change the value of &lt;tt&gt;minThreads&lt;/tt&gt; in the writer thread pool used by replication to 0. This will not eliminate all possible failures due to the glibc bug, but it will eliminate the only one we&apos;ve seen in practice until such time as the bug in glibc is corrected.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1678281">SERVER-56054</key>
            <summary>Change minThreads value for replication writer thread pool to 0</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="lingzhi.deng@mongodb.com">Lingzhi Deng</assignee>
                                    <reporter username="schwerin@mongodb.com">Andy Schwerin</reporter>
                        <labels>
                    </labels>
                <created>Mon, 12 Apr 2021 20:02:02 +0000</created>
                <updated>Sun, 29 Oct 2023 21:54:59 +0000</updated>
                            <resolved>Thu, 29 Apr 2021 01:52:24 +0000</resolved>
                                                    <fixVersion>4.2.15</fixVersion>
                    <fixVersion>4.4.7</fixVersion>
                    <fixVersion>5.0.0-rc0</fixVersion>
                    <fixVersion>4.0.26</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>22</watches>
                                                                                                                <comments>
                            <comment id="4008295" author="JIRAUSER1258850" created="Thu, 19 Aug 2021 00:46:50 +0000"  >&lt;p&gt;Hi Team,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I see the fix is applied in 4.0.26 but the relevant documentation section of 4.0 is not updated. Could you please update the documentation section so we can confidently enable this parameter in 4.0.26.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Venkataraman&lt;/p&gt;</comment>
                            <comment id="3895693" author="xgen-internal-githook" created="Thu, 24 Jun 2021 01:33:06 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;: Change minThreads value for replication writer thread pool to 0&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 136fa52193c342038b3fa35152fa1ed3dee4ee87)&lt;br/&gt;
(cherry picked from commit 4935fe56743814ffc39e9f7d9eebb3648dc2846f)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b9b407313d1afd49e820e2cbb6b75336641dbad9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b9b407313d1afd49e820e2cbb6b75336641dbad9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3867731" author="xgen-internal-githook" created="Wed, 9 Jun 2021 00:37:02 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;: Change minThreads value for replication writer thread pool to 0&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 136fa52193c342038b3fa35152fa1ed3dee4ee87)&lt;br/&gt;
(cherry picked from commit 4935fe56743814ffc39e9f7d9eebb3648dc2846f)&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/56eeca07155e9d5db36934fdbdc36cc118e2c519&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/56eeca07155e9d5db36934fdbdc36cc118e2c519&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3792933" author="xgen-internal-githook" created="Wed, 19 May 2021 19:08:59 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;: Change minThreads value for replication writer thread pool to 0&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 136fa52193c342038b3fa35152fa1ed3dee4ee87)&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4935fe56743814ffc39e9f7d9eebb3648dc2846f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4935fe56743814ffc39e9f7d9eebb3648dc2846f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3751737" author="schwerin" created="Mon, 3 May 2021 14:02:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;, to be thorough, I checked with the head of our release management team. We will not be applying this change to the 3.6 branch because, while it is expected to be quite safe, we have very little tolerance for risk on branches that are past their supported life. As I understand it, we are unlikely to ever release 3.6.24 except to deal with certain critical security issues.&lt;/p&gt;</comment>
                            <comment id="3749352" author="JIRAUSER1258850" created="Fri, 30 Apr 2021 17:37:11 +0000"  >&lt;p&gt;Thanks Andy for the reply and i understand the support policy.. I see 3.6.24 version internally having some fixes already committed. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56126?jql=fixVersion%20%3D%203.6.24&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-56126?jql=fixVersion%20%3D%203.6.24&lt;/a&gt;.&#160;So it would be of great help if we get one last patch on 3.6 train with this fix. i request you to consider this request.&lt;/p&gt;</comment>
                            <comment id="3745548" author="xgen-internal-githook" created="Wed, 28 Apr 2021 23:23:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;: Change minThreads value for replication writer thread pool to 0&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/136fa52193c342038b3fa35152fa1ed3dee4ee87&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/136fa52193c342038b3fa35152fa1ed3dee4ee87&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3742110" author="schwerin" created="Tue, 27 Apr 2021 18:44:06 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;, I&apos;m afraid that per our &lt;a href=&quot;https://www.mongodb.com/support-policy&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support policy&lt;/a&gt;, MongoDB 3.6 is not supported past the end of this month, and there are no more scheduled releases of that product branch. I am hoping to get this fix applied to the 4.0 (BACKPORT-8822), 4.2 (BACKPORT-8821) and 4.4 (BACKPORT-8820) branches once it is tested and committed on the main development branch.&lt;/p&gt;</comment>
                            <comment id="3741706" author="JIRAUSER1258850" created="Tue, 27 Apr 2021 16:51:02 +0000"  >&lt;p&gt;Thank you Andy for the proactive fix from mongo. Could you please also back-port this in 3.6 and provide the image?&lt;/p&gt;</comment>
                            <comment id="3713995" author="schwerin" created="Mon, 12 Apr 2021 20:13:25 +0000"  >&lt;p&gt;If we implement this workaround, I believe we could also add logging &lt;a href=&quot;https://github.com/mongodb/mongo/blob/68f97da9a1762a4d9e53b201d149b60d8383063a/src/mongo/util/concurrency/thread_pool.cpp#L469&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;after this line&lt;/a&gt;. We would log the length of the &lt;tt&gt;_pendingTasks&lt;/tt&gt; queue at this point if the thread woke up after the deadline expired and &lt;tt&gt;_pendingTasks&lt;/tt&gt; were not empty. This would log sometimes when no signal was lost, but given the length of the &lt;tt&gt;maxIdleThreadAge&lt;/tt&gt; value (30 seconds by default), I believe such false positive logging events would be rare.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1634472">SERVER-54805</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="1695299">DOCS-14391</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="1634472">SERVER-54805</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1881608">SERVER-60164</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1979227">SERVER-63402</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1708066">SERVER-56784</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21266"><![CDATA[v4.9]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000rzqfwQAA, 5002K00000uz0O8QAI, 5002K00000vfgicQAA, 5002K00000yVXRNQA4, 5002K000011DlpMQAS]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 13 Apr 2021 07:33:32 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 25 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Added server parameter replWriterMinThreadCount - The minimum number of threads in the thread pool used to apply the oplog with default 0. Secondary oplog application will use up to replWriterThreadCount threads (&lt;a href=&quot;https://docs.mongodb.com/manual/reference/parameters/#mongodb-parameter-param.replWriterThreadCount&quot;&gt;https://docs.mongodb.com/manual/reference/parameters/#mongodb-parameter-param.replWriterThreadCount&lt;/a&gt;). If the newly added replWriterMinThreadCount is less than replWriterThreadCount, the thread pool will timeout idle threads and only keep replWriterMinThreadCount idle threads in the pool. (replWriterMinThreadCount must be less than or equal to replWriterThreadCount)</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 25 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>veramasu@hcl.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz4h6f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyphwn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4858">Repl 2021-05-03</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16943"><![CDATA[Cloud]]></customfieldvalue>
    <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    <customfieldvalue key="16946"><![CDATA[Triage and Release]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz43fj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>