<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:15:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-70155] Add duration of how long an oplog slot is kept open to mongod &quot;Slow query&quot; log lines</title>
                <link>https://jira.mongodb.org/browse/SERVER-70155</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;An oplog hole represents a Timestamp in the oplog an active storage transaction would commit at that is behind the Timestamp in the oplog an already&amp;#45;committed storage transaction wrote their oplog entry at. For example, it is possible for a storage transaction at Timestamp 20 to have committed and for a storage transaction at Timestamp 10 to commit later on in wall&amp;#45;clock time. Oplog readers are prevented from reading beyond the oplog hole to ensure they don&apos;t miss any oplog entries which might still commit. Keeping an oplog hole open for any extended period of wall&amp;#45;clock time can lead to stalls in replication.&lt;/p&gt;

&lt;p&gt;Vectored insert is an example of an operation which pre&amp;#45;allocates Timestamps to write user data and the corresponding oplog entries at. In MongoDB 4.4, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46161&quot; title=&quot;Reduce frequency of yielding during query execution&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46161&quot;&gt;&lt;del&gt;SERVER-46161&lt;/del&gt;&lt;/a&gt; caused the default internal batch size for vectored insert to increase from 64 to 500. This has been seen to lead to higher tail latencies for vectored inserts (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-65054&quot; title=&quot;Avoid slow insert batches blocking replication&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-65054&quot;&gt;SERVER-65054&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Other operations cause oplog holes too. If expensive work is done (e.g. within an OpObserver) after the oplog slot is allocated and prior to the storage transaction committing, then those operations can stall replication too. Introducing some logging to track the time spent would give more insight into these areas and perhaps even be useful to signal on within our performance testing.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2149660">SERVER-70155</key>
            <summary>Add duration of how long an oplog slot is kept open to mongod &quot;Slow query&quot; log lines</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="dianna.hohensee@mongodb.com">Dianna Hohensee</assignee>
                                    <reporter username="max.hirschhorn@mongodb.com">Max Hirschhorn</reporter>
                        <labels>
                    </labels>
                <created>Sun, 2 Oct 2022 12:54:35 +0000</created>
                <updated>Thu, 11 Jan 2024 23:01:39 +0000</updated>
                            <resolved>Mon, 13 Mar 2023 19:23:35 +0000</resolved>
                                                    <fixVersion>7.0.0-rc0</fixVersion>
                    <fixVersion>6.0.13</fixVersion>
                    <fixVersion>5.0.24</fixVersion>
                    <fixVersion>4.4.29</fixVersion>
                                    <component>Replication</component>
                    <component>Write Ops</component>
                                        <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="6005731" author="xgen-internal-githook" created="Thu, 11 Jan 2024 22:47:43 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Dianna Hohensee&apos;, &apos;email&apos;: &apos;dianna.hohensee@mongodb.com&apos;, &apos;username&apos;: &apos;&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70155&quot; title=&quot;Add duration of how long an oplog slot is kept open to mongod &amp;quot;Slow query&amp;quot; log lines&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70155&quot;&gt;&lt;del&gt;SERVER-70155&lt;/del&gt;&lt;/a&gt; Add totalOplogSlotDurationMicros to &apos;Slow query&apos; log msgs and the profiler data, to track how long a write operation holds an oplog slot open before commit&lt;/p&gt;

&lt;p&gt;(cherry picked from commit a7ef4363090042302411a6a28e1fbf4bd18cd3ac)&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: 517acbc392a3bd095a889c4ce672184eb43cb522&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/bb8e21b5f7bb2649ed31d57b6dbc4468541ad80e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/bb8e21b5f7bb2649ed31d57b6dbc4468541ad80e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5975634" author="xgen-internal-githook" created="Fri, 29 Dec 2023 19:15:41 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Dianna Hohensee&apos;, &apos;email&apos;: &apos;dianna.hohensee@mongodb.com&apos;, &apos;username&apos;: &apos;&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70155&quot; title=&quot;Add duration of how long an oplog slot is kept open to mongod &amp;quot;Slow query&amp;quot; log lines&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70155&quot;&gt;&lt;del&gt;SERVER-70155&lt;/del&gt;&lt;/a&gt; Add totalOplogSlotDurationMicros to &apos;Slow query&apos; log msgs and the profiler data, to track how long a write operation holds an oplog slot open before commit&lt;/p&gt;

&lt;p&gt;(cherry picked from commit a7ef4363090042302411a6a28e1fbf4bd18cd3ac)&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: afa8f5824455e023c966cee986a8591ab1eeb049&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9965e5c97cfaeb5d74dee9841eb2effc535d95b7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9965e5c97cfaeb5d74dee9841eb2effc535d95b7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5974456" author="xgen-internal-githook" created="Thu, 28 Dec 2023 23:12:56 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Dianna Hohensee&apos;, &apos;email&apos;: &apos;dianna.hohensee@mongodb.com&apos;, &apos;username&apos;: &apos;&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70155&quot; title=&quot;Add duration of how long an oplog slot is kept open to mongod &amp;quot;Slow query&amp;quot; log lines&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70155&quot;&gt;&lt;del&gt;SERVER-70155&lt;/del&gt;&lt;/a&gt; Add totalOplogSlotDurationMicros to &apos;Slow query&apos; log msgs and the profiler data, to track how long a write operation holds an oplog slot open before commit&lt;/p&gt;

&lt;p&gt;(cherry picked from commit a7ef4363090042302411a6a28e1fbf4bd18cd3ac)&lt;/p&gt;

&lt;p&gt;GitOrigin-RevId: d81c97f81999e99ba248ef096d25a74298df8022&lt;br/&gt;
Branch: v6.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/6077a9ddf0d8cac0a8e10503c8caf257a15efa9c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/6077a9ddf0d8cac0a8e10503c8caf257a15efa9c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5268352" author="xgen-internal-githook" created="Mon, 13 Mar 2023 17:05:26 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Dianna Hohensee&apos;, &apos;email&apos;: &apos;dianna.hohensee@mongodb.com&apos;, &apos;username&apos;: &apos;DiannaHohensee&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70155&quot; title=&quot;Add duration of how long an oplog slot is kept open to mongod &amp;quot;Slow query&amp;quot; log lines&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70155&quot;&gt;&lt;del&gt;SERVER-70155&lt;/del&gt;&lt;/a&gt; Add totalOplogSlotDurationMicros to &apos;Slow query&apos; log msgs and the profiler data, to track how long a write operation holds an oplog slot open before commit&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a7ef4363090042302411a6a28e1fbf4bd18cd3ac&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a7ef4363090042302411a6a28e1fbf4bd18cd3ac&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4916094" author="louis.williams" created="Thu, 20 Oct 2022 15:52:12 +0000"  >&lt;p&gt;For both &lt;a href=&quot;https://github.com/mongodb/mongo/blob/8f524446f17abc2c042aa6ef5f81b70ba6513438/src/mongo/db/repl/topology_coordinator.cpp#L2051&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;appliedOpTime&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/8f524446f17abc2c042aa6ef5f81b70ba6513438/src/mongo/db/repl/replication_info.cpp#L251&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;latestOpTime&lt;/a&gt;, the source of the value is the same. It&apos;s the lastApplied value from the node in question.  Either of these values would be correct as far as I can tell.&lt;/p&gt;

&lt;p&gt;On primaries, this represents the highest OpTime ever committed, so I think the lag calculation as lastApplied - allDurable is correct, regardless of how you obtain it.&lt;/p&gt;</comment>
                            <comment id="4911786" author="louis.williams" created="Wed, 19 Oct 2022 07:21:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yuan.fang%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;yuan.fang@mongodb.com&quot;&gt;yuan.fang@mongodb.com&lt;/a&gt; that looks correct. Can you confirm the full path in serverStatus/FTDC that you&apos;re using to get the &quot;appliedOpTime&quot;? I don&apos;t see that reported in any of the FTDC data that I have locally from a recent build. Are you by any chance using &quot;oplog latestOptime t&quot;? If so, that would be correct.&lt;/p&gt;</comment>
                            <comment id="4879565" author="bruce.lucas@10gen.com" created="Wed, 5 Oct 2022 14:21:08 +0000"  >&lt;p&gt;Thanks, that sounds like exactly what I was looking for. Will open ticket to add that computed metric.&lt;/p&gt;</comment>
                            <comment id="4878868" author="louis.williams" created="Wed, 5 Oct 2022 07:28:43 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas@mongodb.com&quot;&gt;bruce.lucas@mongodb.com&lt;/a&gt; this is the metric that I&apos;ve used in the past to diagnose issues with oplog hole-induced lag, but you make a good point that this metric could also be affected by journaling.&lt;/p&gt;

&lt;p&gt;We do have an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/94fb7dfc8b974f1f5343e7ea394d0d9deedba50e/src/mongo/db/storage/wiredtiger/wiredtiger_server_status.cpp#L101-L102&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;oplog visibility&lt;/a&gt; metric, but the way that it shows up in t2 is not useful:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/406209/406209_Screen+Shot+2022-10-05+at+9.27.40+AM.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;So I think it would make sense to add a metric that calculates the lag between the oplog visbility timestamp and the wall time.&lt;/p&gt;</comment>
                            <comment id="4876964" author="bruce.lucas@10gen.com" created="Tue, 4 Oct 2022 15:59:55 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams@mongodb.com&quot;&gt;louis.williams@mongodb.com&lt;/a&gt;, just to confirm, are oplog holes the &lt;em&gt;only&lt;/em&gt; thing that can cause lastDurable to lag? For example, would a storage bottleneck (or stall) in the journal possibly cause that metric to lag? If so, is there a metric that relates &lt;em&gt;only&lt;/em&gt; to oplog holes, or could one be added?&lt;/p&gt;</comment>
                            <comment id="4876049" author="louis.williams" created="Tue, 4 Oct 2022 13:02:04 +0000"  >&lt;p&gt;The &quot;rs lastDurable wall time lag&quot; should be the serverStatus metric we use that tells us whether there are operations holding oplog holes open for long periods of time.&lt;/p&gt;</comment>
                            <comment id="4876017" author="bruce.lucas@10gen.com" created="Tue, 4 Oct 2022 12:48:31 +0000"  >&lt;p&gt;I wonder if there&apos;s something that could be added to serverStatus (and therefore FTDC) also - or is there already a metric that clearly indicates the component of lag caused by oplog holes?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="2287103">DOCS-15960</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2533860">SERVER-84449</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2534841">SERVER-84467</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2280902">SERVER-74604</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2011730">SERVER-65054</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="406209" name="Screen Shot 2022-10-05 at 9.27.40 AM.png" size="135282" author="louis.williams@mongodb.com" created="Wed, 5 Oct 2022 07:28:28 +0000"/>
                            <attachment id="408778" name="oplog_visbility_lag_example.png" size="138080" author="yuan.fang@mongodb.com" created="Mon, 17 Oct 2022 18:34:08 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="26553"><![CDATA[v7.2]]></customfieldvalue>
    <customfieldvalue key="25578"><![CDATA[v7.0]]></customfieldvalue>
    <customfieldvalue key="23470"><![CDATA[v6.0]]></customfieldvalue>
    <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 4 Oct 2022 12:48:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>There is a new field called totalOplogSlotDurationMicros logged in &amp;quot;Slow query&amp;quot; messages and written to the database profiler collection, when the value is non-zero.&lt;br/&gt;
&lt;br/&gt;
As a bit of background, mongod supports parallel writes, but there is a small serialization critical section where writes are handed commit timestamps with which to commit the storage engine writes. Once write operations have commit timestamps, they can commit in any order: out-of-order writes.&lt;br/&gt;
&lt;br/&gt;
For example,&lt;br/&gt;
&lt;br/&gt;
writeA receives Timestamp1,&lt;br/&gt;
writeB receives Timestamp2,&lt;br/&gt;
writeC receives Timestamp3,&lt;br/&gt;
&lt;br/&gt;
then say writeB commits first at Timestamp2. Replication will not proceed to replicate writeB until after writeA commits. writeA is expected to create an oplog entry with Timestamp1 before replication will copy the oplog to secondaries replica set members. This is the idea of an oplog slot. If writeA were to be delayed somehow in committing -- say CPU thread management, or getting stuck in MDB someplace -- then that would delay replication. The new totalOplogSlotDurationMicros value represents the time between a write operation getting a commit timestamp and actually committing, and can help surface issues where operations are committing slowly and slowing down replication.</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2884</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1cgif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1bhgb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="6773">Execution Team 2023-03-06</customfieldvalue>
    <customfieldvalue id="6816">Execution Team 2023-03-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1c2nr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>