<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:54:51 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-62340] Tenant Migration can lead to leakage of &quot;TenantMigrationBlockerAsync&quot; threads.</title>
                <link>https://jira.mongodb.org/browse/SERVER-62340</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While investigating the&#160;BF, it revealed that the tenant migration donor code can lead to leakage of &quot;TenantMigrationBlockerAsync&quot; threads. Consider the below scenario:&lt;/p&gt;

&lt;p&gt;1) Donor starts migration for tenant &lt;tt&gt;foo&lt;/tt&gt;.&lt;br/&gt;
2) Donor secondary on receiving the state doc with &quot;kAbortingIndexBuilds&quot; state, it creates an access blocker for tenant &lt;tt&gt;foo&lt;/tt&gt; and &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_donor_op_observer.cpp#L56-L63&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;add&lt;/a&gt; it to the blocker registry. &lt;b&gt;Now, the shared reference count on the donor access blocker will be 1.&lt;/b&gt;&lt;br/&gt;
3) Donor secondary receives a find command for tenant &lt;tt&gt;foo&lt;/tt&gt;.&lt;br/&gt;
4) Find request calls &lt;tt&gt;checkIfCanReadOrBlock()&lt;/tt&gt;  and asynchronously waits on the &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp#L159&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;canRead promise&lt;/a&gt; to be fulfilled and &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp#L189-L191&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;captures&lt;/a&gt; the tenant&apos;s  donor access blocker as shared pointer. &lt;b&gt;Now, the shared reference count on the donor access blocker will be 2.&lt;/b&gt;&lt;br/&gt;
5) Donor primary durably commits the migration for tenant &quot;foo&quot; and marks the state document as garbage collectible.&lt;br/&gt;
6) Donor secondary on receiving the donor state doc with &apos;expireAt&apos; set, it would &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_donor_op_observer.cpp#L163&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;fulfill the canRead promise&lt;/a&gt; and &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_donor_op_observer.cpp#L172&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;remove&lt;/a&gt; the  donor access blocker for tenant &lt;tt&gt;foo&lt;/tt&gt;  from the blocker registry. &lt;b&gt;Now, the shared reference count on the donor access blocker will reduce to 1&lt;/b&gt;&lt;br/&gt;
7) On canRead promise fulfillment, we run &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp#L191-L232&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; continuation chain on &lt;br/&gt;
&lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_donor_access_blocker.h#L346&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;_asyncBlockingOperationsExecutor&lt;/a&gt;, backed up the thread pool &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_access_blocker_executor.cpp#L57&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;TenantMigrationBlockerAsyncThreadPool&lt;/a&gt;. Once the work in the chain is completed, it will release all the &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp#L191&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;captured resources&lt;/a&gt;.  &lt;b&gt;This will lead the shared reference count on the donor access blocker to decrement by 1&lt;/b&gt;, i.e, number of shared owners will now be 0.&lt;/p&gt;

&lt;p&gt;This results in calling of &lt;tt&gt;TenantMigrationDonorAccessBlocker&lt;/tt&gt; destructor, which in turn results in calling of &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/db/repl/tenant_migration_donor_access_blocker.h#L346&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;_asyncBlockingOperationsExecutor&lt;/tt&gt;&apos;s&lt;/a&gt; destructor (This thread pool executor is shared by all donor access blockers and is destroyed when no access blockers exist),   that makes the executor to &lt;a href=&quot;https://github.com/10gen/mongo/blob/9ec1e6e58e94d52d2f6a9bc167ff939118aa5134/src/mongo/executor/thread_pool_task_executor.cpp#L143-L144&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shutdown and waits for executor to join&lt;/a&gt;. &lt;b&gt;But, the executor join() is blocked waiting for current &quot;TenantMigrationBlockerAsync-X&quot; thread  to join and the current &quot;TenantMigrationBlockerAsync-X&quot; thread is  waiting for executor _join() to complete&lt;/b&gt;, leading to self-deadlock and leakage of &quot;TenantMigrationBlockerAsync&quot; threads.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Note: The same problem exist on the recipient side as well.&lt;/b&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1958055">SERVER-62340</key>
            <summary>Tenant Migration can lead to leakage of &quot;TenantMigrationBlockerAsync&quot; threads.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="didier.nadeau@mongodb.com">Didier Nadeau</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Tue, 4 Jan 2022 14:43:22 +0000</created>
                <updated>Sun, 29 Oct 2023 21:44:34 +0000</updated>
                            <resolved>Fri, 14 Jan 2022 16:10:56 +0000</resolved>
                                                    <fixVersion>5.3.0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="4293265" author="xgen-internal-githook" created="Fri, 14 Jan 2022 15:59:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Didier Nadeau&apos;, &apos;email&apos;: &apos;didier.nadeau@mongodb.com&apos;, &apos;username&apos;: &apos;nadeaudi&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-62340&quot; title=&quot;Tenant Migration can lead to leakage of &amp;quot;TenantMigrationBlockerAsync&amp;quot; threads.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-62340&quot;&gt;&lt;del&gt;SERVER-62340&lt;/del&gt;&lt;/a&gt; Move executor to TenantAccessBlockerRegistry to prevent destruction issues&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a22829787f370bd4d7e4fd767517dd616169638e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a22829787f370bd4d7e4fd767517dd616169638e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4289128" author="esha.maharishi@10gen.com" created="Wed, 12 Jan 2022 19:17:19 +0000"  >&lt;p&gt;Sounds good. Just a note that I&apos;m not sure if first bullet would be an issue in practice. I didn&apos;t get to mention this on Zoom, but there are existing places where we block user threads and unblock them all at once, such as on exiting the sharding migration critical section.&lt;/p&gt;</comment>
                            <comment id="4289088" author="JIRAUSER1262830" created="Wed, 12 Jan 2022 19:05:57 +0000"  >&lt;p&gt;Following a discussion with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;&#160; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=suganthi.mani&quot; class=&quot;user-hover&quot; rel=&quot;suganthi.mani&quot;&gt;suganthi.mani&lt;/a&gt; , we decided to go ahead with Suganthi&apos;s idea to move the ownership of the ThreadPool to `TenantMigrationAccessBlockerRegistry` and make it a `shared_ptr`. As this means it will always exist, we will set the `minThread` to 0 so that, when there is no migration, no threads exist to remove impact on non-serverless instances.&lt;/p&gt;

&lt;p&gt;We decided not to go ahead and integrate the `checkIfCanReadOrBlock`&apos;s future into the caller&apos;s future chain for two reasons :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;In Synchronous mode (one thread per client operation) there can be a very large number of threads that would wait on the future at the same time. In turns this could lead to an overload as all these threads would be unblocked and try to do some work at the same time. See comment &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-52783?focusedCommentId=3541412&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-3541412&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;here&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Currently the future is interruptible using the opCtx (`wait(opCtx)`). There does not seem to be a clean and straightforward way to preserve the interruptibility if we integrate the future into the caller&apos;s chain (i.e. `future.thenRunOn(callerExecutor).then(...)`)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="4286906" author="suganthi.mani" created="Tue, 11 Jan 2022 21:20:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=didier.nadeau&quot; class=&quot;user-hover&quot; rel=&quot;didier.nadeau&quot;&gt;didier.nadeau&lt;/a&gt; To my understanding, there will be an active POS executor only on primary. But, we maintain access blocker + it&apos;s executor both on primary and secondaries. So, I think probably the idea#2 (Retrieve the access_blocker&apos;s executor and run a simple lambda on the POS&apos;s executor) won&apos;t work.&lt;/p&gt;</comment>
                            <comment id="4286796" author="JIRAUSER1262830" created="Tue, 11 Jan 2022 20:38:39 +0000"  >&lt;p&gt;Some ideas :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Finish the continuation on a different executor that does nothing but capture the access_blocker. This ensures the destructor is not run on the access_blocker&apos;s executor, preventing the deadlock.
	&lt;ul&gt;
		&lt;li&gt;One possible issue problem with this would be that a very large amount of continuations are unblocked at the same time (multiple operations waiting on the future) leading to an executor overload &lt;a href=&quot;https://github.com/10gen/mongo/blob/dffa83e682aa899e0e3ce6e93831a3cc26dc6639/src/mongo/db/repl/tenant_migration_access_blocker_executor.h#L46-L52&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;GitHub comment&lt;/a&gt;&lt;/li&gt;
		&lt;li&gt;This could be the main POS cleanup executor ?&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Retrieve the access_blocker&apos;s executor and run a simple lambda on the POS&apos;s executor. The lambda would be :&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;return&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; AsyncTry([accessBlockerExecutorSharedPtrAnchor] () {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    .until([accessBlockerWeakPtr]() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;return&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; !!accessBlockerWeakPtr.lock();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    .withDelayBetweenIterations(miliseconds(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    .on(**posExecutor&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;Possible issue : handling shutdown of the POS executor ?&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="4284198" author="esha.maharishi@10gen.com" created="Mon, 10 Jan 2022 20:44:08 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=didier.nadeau&quot; class=&quot;user-hover&quot; rel=&quot;didier.nadeau&quot;&gt;didier.nadeau&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matt.broadstone&quot; class=&quot;user-hover&quot; rel=&quot;matt.broadstone&quot;&gt;matt.broadstone&lt;/a&gt; to triage along with Implement Split work tomorrow.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 10 Jan 2022 20:44:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 3 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 3 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>didier.nadeau@mongodb.com</customfieldvalue>
            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0fuvb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzz1c7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5671">Server Serverless 2022-01-24</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0fh0n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>