<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:02:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-43155] Queries which exceed maxTimeMS may return NetworkInterfaceExceededTimeLimit</title>
                <link>https://jira.mongodb.org/browse/SERVER-43155</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In &lt;a href=&quot;https://github.com/mongodb/mongo/blob/1d21a5b64114fe777ee879713552048c0fbba631/src/mongo/s/client/shard_remote.cpp#L300-L301&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shard_remote&lt;/a&gt;, the maxTimeMS of a query is used as a request timeout. If the time limit is exceeded by the network layer, &quot;NetworkInterfaceExeededTimeLimit&quot; will be returned instead of MaxTimeMSExpired. (e.g. &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bfedc310559048d88747901fe62271bb1f62a90e/src/mongo/executor/network_interface_tl.cpp#L383&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bfedc310559048d88747901fe62271bb1f62a90e/src/mongo/executor/network_interface_tl.cpp#L415&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here).&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="917014">SERVER-43155</key>
            <summary>Queries which exceed maxTimeMS may return NetworkInterfaceExceededTimeLimit</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="backlog-server-servicearch">Backlog - Service Architecture</assignee>
                                    <reporter username="ian.boros@mongodb.com">Ian Boros</reporter>
                        <labels>
                    </labels>
                <created>Wed, 4 Sep 2019 18:22:07 +0000</created>
                <updated>Sun, 29 Oct 2023 22:17:28 +0000</updated>
                            <resolved>Mon, 18 Jul 2022 14:39:50 +0000</resolved>
                                                    <fixVersion>5.0.11</fixVersion>
                    <fixVersion>6.1.0-rc0</fixVersion>
                                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="4718379" author="xgen-internal-githook" created="Mon, 1 Aug 2022 15:55:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Daniel Morilha&apos;, &apos;email&apos;: &apos;daniel.morilha@mongodb.com&apos;, &apos;username&apos;: &apos;daniel-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43155&quot; title=&quot;Queries which exceed maxTimeMS may return NetworkInterfaceExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43155&quot;&gt;&lt;del&gt;SERVER-43155&lt;/del&gt;&lt;/a&gt; Queries which exceed `maxTimeMS` may return `NetworkInterfaceExceededTimeLimit`&lt;/p&gt;

&lt;p&gt;After careful evaluation and research the service-arch team settled into&lt;br/&gt;
adding a new argument into connection pool&apos;s interface allowing their&lt;br/&gt;
users to specify a custom timeout status in case the pool fails to&lt;br/&gt;
deliver a connection within a specified time.&lt;/p&gt;

&lt;p&gt;This timeout status now competes with `NetworkInterfaceExceededTimeLimit`&lt;br/&gt;
which is now returned either when the Connection Pool Controller is rather&lt;br/&gt;
used to compute the timeout or when no custom timeout status code is&lt;br/&gt;
specified.&lt;/p&gt;

&lt;p&gt;This commit also:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Refactors portions of `ConnectionPoolTest` to reflect changes in the&lt;br/&gt;
API.&lt;/li&gt;
	&lt;li&gt;Adds `jstests/sharding/max_time_ms_connection_pool.js` integration&lt;br/&gt;
test to ensure different timeouts return different statuses.&lt;/li&gt;
	&lt;li&gt;Fixes small glitch in `jstests/core/tnxs/many_txns.js` where&lt;br/&gt;
`NetworkInterfaceExceededTimeLimit` might be validly returned under rare&lt;br/&gt;
resource intense conditions and fail the test, reported as BF-20554.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;(cherry picked from commit 3fd2a61085f8c72a13d1bc00ad95c460c69eb8a8)&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/81cd7f7f3d69937a7c7b2a9ee58ac326ce059fe3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/81cd7f7f3d69937a7c7b2a9ee58ac326ce059fe3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4642616" author="xgen-internal-githook" created="Mon, 27 Jun 2022 17:43:43 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Varun Ravichandran&apos;, &apos;email&apos;: &apos;varun.ravichandran@mongodb.com&apos;, &apos;username&apos;: &apos;varunravi98&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43155&quot; title=&quot;Queries which exceed maxTimeMS may return NetworkInterfaceExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43155&quot;&gt;&lt;del&gt;SERVER-43155&lt;/del&gt;&lt;/a&gt; Queries which exceed `maxTimeMS` may return `NetworkInterfaceExceededTimeLimit`&lt;/p&gt;

&lt;p&gt;After careful evaluation and research the service-arch team settled into&lt;br/&gt;
adding a new argument into connection pool&apos;s interface allowing their&lt;br/&gt;
users to specify a custom timeout status in case the pool fails to&lt;br/&gt;
deliver a connection within a specified time.&lt;/p&gt;

&lt;p&gt;This timeout status now competes with `NetworkInterfaceExceededTimeLimit`&lt;br/&gt;
which is now returned either when the Connection Pool Controller is rather&lt;br/&gt;
used to compute the timeout or when no custom timeout status code is&lt;br/&gt;
specified.&lt;/p&gt;

&lt;p&gt;This commit also:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Refactors portions of `ConnectionPoolTest` to reflect changes in the&lt;br/&gt;
API.&lt;/li&gt;
	&lt;li&gt;Adds `jstests/sharding/max_time_ms_connection_pool.js` integration&lt;br/&gt;
test to ensure different timeouts return different statuses.&lt;/li&gt;
	&lt;li&gt;Fixes small glitch in `jstests/core/tnxs/many_txns.js` where&lt;br/&gt;
`NetworkInterfaceExceededTimeLimit` might be validly returned under rare&lt;br/&gt;
resource intense conditions and fail the test, reported as BF-20554.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3fd2a61085f8c72a13d1bc00ad95c460c69eb8a8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3fd2a61085f8c72a13d1bc00ad95c460c69eb8a8&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="4562943" author="JIRAUSER1263891" created="Fri, 20 May 2022 17:28:43 +0000"  >&lt;p&gt;After an hour-long discussion with different stakeholders (reviewers to the proposed change) requirements change and so did my understanding of what is expected from this ticket and where to draw the boundary line in terms of responsibilities.&lt;/p&gt;

&lt;p&gt;George pointed out for the existence of &lt;em&gt;RemoteCommandRequest::timeoutCode&lt;/em&gt; and how that is used in order to override the expected user-defined timeout. While I&apos;ve noticed that it never occurred to me that that would be the case for this given instance. It was then agreed &lt;em&gt;ConnectionPool&lt;/em&gt; is ambiguous as it may override the user-defined timeout duration based on internal heuristics still returning the same error code, namely &quot;NetworkInterfaceTimeLimitException&quot; and how that should be addressed by both &lt;em&gt;ConnectionPool&lt;/em&gt; and &lt;em&gt;NetworkInterfaceTL&lt;/em&gt; appropriately, the user-specified exception should be returned iff the user-specified deadline expired.&lt;/p&gt;

&lt;p&gt;It was also reiterated that given errors are returned to the original remote command requester, the requester might be at a better place to treat those errors as they please as moving this sort of logic into parts that are supposed to be generic defeats the very same genericity.&lt;/p&gt;

&lt;p&gt;It was also evaluated that the errors occurring in BF-20554 while seem to overlap with this request have nothing to deal with what has been asked here. There, the problem seems to be brittle tests which rely on a specific error being reported when transactions conflict may fail while running into a resource constrained environment. Another specific change is expected for that.&lt;/p&gt;

&lt;p&gt;We also had a chance to evaluate the feseability to retire &lt;em&gt;OperationContext&lt;/em&gt; --&amp;gt; &lt;em&gt;storeMaxTimeMS&lt;/em&gt;, &lt;em&gt;restoreMaxTimeMS&lt;/em&gt; and &lt;em&gt;{&lt;/em&gt;}maxTime{_} as it may not being used any longer.&lt;/p&gt;

&lt;p&gt;Code changes will follow...&lt;/p&gt;</comment>
                            <comment id="4529485" author="shane.harvey" created="Thu, 5 May 2022 18:51:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn@mongodb.com&quot;&gt;max.hirschhorn@mongodb.com&lt;/a&gt; brought to my attention some subtlety that I missed in the previous question. If the client sends maxTimeMS 60 and the server only waits 30 seconds before returning MaxTimeMSExpired, that behavior seems unexpected. In this case it should be fine to return a non-MaxTimeMSExpired error code since the time limit hasn&#8217;t actually expired. Ideally MaxTimeMSExpired would only be returned when maxTimeMS has actually expired. &lt;/p&gt;

&lt;p&gt;Note that the opposite (ie &quot;when maxTimeMS expires the server always returns MaxTimeMSExpired&quot;) isn&apos;t necessarily true. It&apos;s expected that the server may return other error codes. For example the server could be in the process of returning a shutdown error when maxTimeMS expires in which case returning ShutdownInProgress is fine.&lt;/p&gt;</comment>
                            <comment id="4517419" author="shane.harvey" created="Fri, 29 Apr 2022 20:08:34 +0000"  >&lt;p&gt;It would be great for the server to return MaxTimeMSExpired instead of NetworkInterfaceExceededTimeLimit. Clients expect the MaxTimeMSExpired error to be returned when a request exceeds MaxTimeMS.&lt;/p&gt;</comment>
                            <comment id="4516751" author="JIRAUSER1263891" created="Fri, 29 Apr 2022 16:24:38 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shane.harvey%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;shane.harvey@mongodb.com&quot;&gt;shane.harvey@mongodb.com&lt;/a&gt;, after discussing this SERVER ticket with both Max and Sam we&apos;ve identified two scenarios which might concern drivers, they are:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;if a command sets MaxTimeMS of 60 seconds, the connection pool might return &lt;a href=&quot;https://github.com/10gen/mongo/blob/1b80d1290c1c974a8cbdf8c22ff7b187179db3fd/src/mongo/executor/network_interface_tl.cpp#L590-L605&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;NetworkInterfaceExceededTimeLimit&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;if a command sets MaxTimeMS of 60 seconds, but the connection pool has a &lt;a href=&quot;https://github.com/10gen/mongo/blob/master/src/mongo/executor/connection_pool.cpp#L680-L683&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;refreshTimeout&lt;/a&gt; of 30, it might return the same NetworkInterfaceExceededTimeLimit.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This exception only tells a connection wasn&apos;t available within the requested timeframe. The underlying reason gets hidden away from the requester and we believe it should be re-tried.&lt;/p&gt;

&lt;p&gt;The easiest approach would be to have it rewritten to &lt;em&gt;MaxTimeMSExpired&lt;/em&gt; at either the client or the sharding layer.&lt;/p&gt;

&lt;p&gt;I hope you may answer in terms of the client expectation, if it would make sense to rewrite one exception into the other, and if so, if that could be done even if the specified timeout was shorter than what the command specified or if the executor should add a retry logic itself.&lt;/p&gt;

&lt;p&gt;Please let me know if all of that makes sense,&lt;/p&gt;</comment>
                            <comment id="4507449" author="JIRAUSER1263891" created="Tue, 26 Apr 2022 17:33:47 +0000"  >&lt;p&gt;After discussing this with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=amirsaman.memaripour%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;amirsaman.memaripour@mongodb.com&quot;&gt;amirsaman.memaripour@mongodb.com&lt;/a&gt;, &lt;em&gt;MaxTimeMSExpired&lt;/em&gt; only relates with client operations.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;em&gt;executor&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;mongo::executor::ConnectionPool::SpecificPool::getConnection&lt;/li&gt;
	&lt;li&gt;mongo::executor::ConnectionPool::get&lt;/li&gt;
	&lt;li&gt;mongo::executor::NetworkInterfaceTL::startCommand&lt;/li&gt;
	&lt;li&gt;mongo::executor::ThreadPoolTaskExecutor::scheduleRemoteCommandOnAny&lt;/li&gt;
	&lt;li&gt;mongo::executor::ShardingTaskExecutor::scheduleRemoteCommandOnAny&lt;/li&gt;
	&lt;li&gt;mongo::executor::TaskExecutor::scheduleRemoteCommand&lt;/li&gt;
	&lt;li&gt;&lt;em&gt;client&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;mongo::RemoteCommandRetryScheduler::_schedule_inlock&lt;/li&gt;
	&lt;li&gt;mongo::RemoteCommandRetryScheduler::startup&lt;/li&gt;
	&lt;li&gt;mongo::Fetcher::schedule&lt;/li&gt;
	&lt;li&gt;&lt;em&gt;shard/client&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;mongo::ShardRemote::_runExhaustiveCursorCommand&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Based on the above stack trace I think the exception should go on RemoteCommandRetryScheduler which is a client&apos;s concern.&lt;/p&gt;</comment>
                            <comment id="4478726" author="JIRAUSER1263891" created="Tue, 12 Apr 2022 22:48:50 +0000"  >&lt;p&gt;PR : &lt;a href=&quot;https://github.com/10gen/mongo/pull/4597&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/10gen/mongo/pull/4597&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4476375" author="JIRAUSER1261330" created="Tue, 12 Apr 2022 11:28:34 +0000"  >&lt;p&gt;I have recently seen &lt;a href=&quot;https://jira.mongodb.org/browse/BF-24814&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;BF-24814&lt;/a&gt; that seems to be related to this. It happens on the &lt;/p&gt;
{taskExecutorPoolSize: 4}
&lt;p&gt; build variant. I will temporarily allow the NetworkInterfaceExceededTimeLimit code and leave a TODO for this ticket.&lt;/p&gt;</comment>
                            <comment id="4475143" author="JIRAUSER1263891" created="Mon, 11 Apr 2022 22:47:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43155&quot; title=&quot;Queries which exceed maxTimeMS may return NetworkInterfaceExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43155&quot;&gt;&lt;del&gt;SERVER-43155&lt;/del&gt;&lt;/a&gt; tells a compelling story for BF-20554. The BF which stills&lt;br/&gt;
occurs for both 5.0 and at the tip of the repository, triggers&lt;br/&gt;
`deferredStateUpdateFunc`, a lambda inside&lt;br/&gt;
`src/mongo/executor/connection_pool.cpp` which sweeps through `requests`&lt;br/&gt;
checking their `Date_t` expiration time and possibly assigning the&lt;br/&gt;
`NetworkInterfaceExceededTimeLimit` error to its associated promise.&lt;/p&gt;

&lt;p&gt;The error surfaces while running `jstests/core/txns/many_txns.js` on a&lt;br/&gt;
`rhel80-small` instance. The test, which runs concurrently with other tests,&lt;br/&gt;
stresses the host by committing many parallel machines, and then querying&lt;br/&gt;
for them.&lt;/p&gt;

&lt;p&gt;Giving the load, query commands may exceed their `maxTimeMS` and may get&lt;br/&gt;
`NetworkInterfaceExceedTimeLimit` mistakenly assigned as a result.&lt;/p&gt;

&lt;p&gt;It is worth mentioning this could not be reproduced locally.&lt;/p&gt;</comment>
                            <comment id="4466546" author="max.hirschhorn@10gen.com" created="Fri, 8 Apr 2022 04:03:08 +0000"  >&lt;p&gt;Reopening this ticket because the MaxTimeMSExpired error code (50) is semantically meaningful to drivers (e.g. ExecutionTimeout exception type, behavior of adding label UnknownTransactionCommitResult to commitTransaction with maxTimeMS) and must be used for errors related to using maxTimeMS. See also &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35031&quot; title=&quot;ExceededTimeLimit (50) is reported in situations unrelated to maxTimeMS&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35031&quot;&gt;&lt;del&gt;SERVER-35031&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="4375662" author="JIRAUSER1262719" created="Thu, 24 Feb 2022 21:55:19 +0000"  >&lt;p&gt;We haven&#8217;t heard back from you for at least one calendar year, so this issue is being closed. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;</comment>
                            <comment id="2412945" author="david.storch" created="Mon, 9 Sep 2019 15:45:26 +0000"  >&lt;p&gt;The query team believes that this should be triaged by the service architecture team.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10620">
                    <name>Issue split</name>
                                            <outwardlinks description="split to">
                                        <issuelink>
            <issuekey id="2040712">SERVER-66260</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2038637">SERVER-66162</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2413471">SERVER-79888</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1975135">SERVER-63181</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2023010">SERVER-65472</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2080922">SERVER-67722</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1975135">SERVER-63181</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25132"><![CDATA[Service Arch]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 9 Sep 2019 15:45:26 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 27 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<a href='https://jira.mongodb.org/browse/PM-1620'>PM-1620</a>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 27 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>17.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-servicearch</customfieldvalue>
            <customfieldvalue>daniel.morilha@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>ian.boros@mongodb.com</customfieldvalue>
            <customfieldvalue>lauren.lewis@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>rui.liu@mongodb.com</customfieldvalue>
            <customfieldvalue>shane.harvey@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvojy7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvdacv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="6095">Service Arch 2022-04-18</customfieldvalue>
    <customfieldvalue id="6136">Service Arch 2022-05-02</customfieldvalue>
    <customfieldvalue id="6137">Service Arch 2022-05-16</customfieldvalue>
    <customfieldvalue id="6263">Service Arch 2022-05-30</customfieldvalue>
    <customfieldvalue id="6267">Service Arch 2022-06-13</customfieldvalue>
    <customfieldvalue id="6283">Service Arch 2022-06-27</customfieldvalue>
    <customfieldvalue id="6303">Service Arch 2022-07-11</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvo67j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>