<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:52 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39383] Speculative majority change stream update lookup on secondary may return pre-image </title>
                <link>https://jira.mongodb.org/browse/SERVER-39383</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently speculative majority change streams on secondary read from the local snapshot and then, after executing, wait for the most recent lastApplied optime to majority commit. For update lookup queries, reading at a local snapshot means that nothing prevents them from reading in the middle of secondary batch application, since majority read concern levels by default do &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e12dcc7fdbdb44fb7806dfb42a49bd740f361d82/src/mongo/db/db_raii.cpp#L206-L211&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;not read from the current lastApplied&lt;/a&gt; timestamp. Since we write out oplog entries before we apply them on secondaries, this means that an update lookup query may  return a document version &lt;em&gt;earlier&lt;/em&gt; than its associated change event. To fix this, we should instead have speculative change stream oplog queries and update lookup queries both read from the lastApplied timestamp. After executing, they can wait for this read timestamp to majority commit. This will ensure that update lookup queries always read data at least as new as their associated change event.&lt;/p&gt;</description>
                <environment></environment>
        <key id="684317">SERVER-39383</key>
            <summary>Speculative majority change stream update lookup on secondary may return pre-image </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Tue, 5 Feb 2019 20:55:03 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:25 +0000</updated>
                            <resolved>Mon, 18 Mar 2019 21:34:38 +0000</resolved>
                                                    <fixVersion>4.1.10</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2184403" author="xgen-internal-githook" created="Mon, 18 Mar 2019 21:34:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39383&quot; title=&quot;Speculative majority change stream update lookup on secondary may return pre-image &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39383&quot;&gt;&lt;del&gt;SERVER-39383&lt;/del&gt;&lt;/a&gt; Add a test for speculative majority change stream secondary reads during batch application&lt;/p&gt;

&lt;p&gt;This commit adds an integration test to verify that speculative majority change stream reads do not return incorrect results when reading concurrently with secondary batch application. The goal is to ensure that, due to the changes from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39356&quot; title=&quot;Change stream updateLookup queries with speculative majority may return uncommitted data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39356&quot;&gt;&lt;del&gt;SERVER-39356&lt;/del&gt;&lt;/a&gt;, these reads will read from the most recent lastApplied timestamp on secondaries.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c8120ddaf8a8bd9da9c8095165a4df485d5a58c9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c8120ddaf8a8bd9da9c8095165a4df485d5a58c9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2180134" author="william.schultz" created="Wed, 13 Mar 2019 19:44:06 +0000"  >&lt;p&gt;Code review URL:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://mongodbcr.appspot.com/453960002/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mongodbcr.appspot.com/453960002/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2177648" author="tess.avitabile" created="Mon, 11 Mar 2019 20:27:37 +0000"  >&lt;p&gt;Based on the above, I would agree that this bug does not exist.&lt;/p&gt;</comment>
                            <comment id="2177213" author="william.schultz" created="Mon, 11 Mar 2019 16:05:57 +0000"  >&lt;p&gt;Ok, so I have one additional finding which is relevant. When we are not doing document post image lookups, change streams will only read the oplog. When doing update lookups, though, we will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b21647a5e19e5980186e57f77be0b1e0ef92a087/src/mongo/db/pipeline/document_source_lookup_change_post_image.cpp#L124-L130&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;call into&lt;/a&gt; &lt;tt&gt;MongoInterfaceStandalone&lt;/tt&gt; to find a document&apos;s post image. The &lt;tt&gt;MongoInterfaceStandalone::lookupSingleDocument&lt;/tt&gt; method &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fcbc0c9c936c83612545ee5873b649854e4b5e57/src/mongo/db/pipeline/process_interface_standalone.cpp#L416&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;makes a call&lt;/a&gt; into &lt;tt&gt;MongoInterfaceStandalone::_getCollectionDefaultCollator&lt;/tt&gt;, which, as it turns out, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/fcbc0c9c936c83612545ee5873b649854e4b5e57/src/mongo/db/pipeline/process_interface_standalone.cpp#L584&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;takes a MODE_IS collection lock&lt;/a&gt; using the AutoGetCollection interface. AutoGetCollection &lt;a href=&quot;https://github.com/mongodb/mongo/blob/acddc7f35f0373ccb2e8fe9d45f42304b2b74f95/src/mongo/db/catalog_raii.cpp#L66-L138&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;doesn&apos;t include any special logic&lt;/a&gt; to have operations avoid conflicting with secondary batch application i.e. they will block against the PBWM. I think this should imply that update lookup queries done locally will never be able to run concurrently with batch application i.e. they should only be able to read strictly before or after application of a batch. The change stream oplog read should be able to read while the batch is being applied, but once it gets to the update lookup query, it will block until the batch is complete and the PBWM is released. This should mean that, at least with the current implementation of update lookup queries, it shouldn&apos;t be possible for them to return a pre-image of their associated change event. &lt;/p&gt;</comment>
                            <comment id="2175860" author="william.schultz" created="Fri, 8 Mar 2019 16:21:07 +0000"  >&lt;p&gt;Your point is valid. I think I need to alter my view a bit. Above I said that the visibility point on secondaries could never be advanced in the middle of batch application, but I think that is not accurate, as you said. This &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7f620154e595d2c1e6c7af79fc62070ced3bb941/src/mongo/db/storage/wiredtiger/wiredtiger_oplog_manager.cpp#L168-L169&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;condition variable&lt;/a&gt; in the oplog journal thread loop is only signalled by &lt;tt&gt;WiredTigerOplogManager::triggerJournalFlush&lt;/tt&gt;, which is only called in two places as referenced above. It should certainly be possible, though, for that condition variable to be woken up at any time, or to be signalled from inside &lt;tt&gt;WiredTigerRecoveryUnit::_txnClose&lt;/tt&gt; &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7f620154e595d2c1e6c7af79fc62070ced3bb941/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L412&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. I had trouble reproing this case perhaps because there was nothing to call &lt;tt&gt;triggerJournalFlush()&lt;/tt&gt; from within &lt;tt&gt;_txnClose&lt;/tt&gt;. It should be possible for that to happen, though (or for the CV to be spuriously woken up). I will need to work more to come up with a clean repro of this. &lt;/p&gt;</comment>
                            <comment id="2175677" author="tess.avitabile" created="Fri, 8 Mar 2019 14:26:29 +0000"  >&lt;blockquote&gt;
&lt;p&gt;So, this means that the oplog visibility point on secondaries should never advance in the middle of batch application.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I&apos;m confused, since this was not my understanding of the system. The fact that oplog readers can read a batch that has not yet been applied was what caused the invariant failure on secondaries in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38499&quot; title=&quot;Preparing transaction fails and triggers invariant if chosen timestamp is not greater than WiredTiger&amp;#39;s latest active read timestamp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38499&quot;&gt;&lt;del&gt;SERVER-38499&lt;/del&gt;&lt;/a&gt;. We considered fixing &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38499&quot; title=&quot;Preparing transaction fails and triggers invariant if chosen timestamp is not greater than WiredTiger&amp;#39;s latest active read timestamp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38499&quot;&gt;&lt;del&gt;SERVER-38499&lt;/del&gt;&lt;/a&gt; by not updating the all-committed until the end of batch application but decided not to do so, since it would slow majority writes.&lt;/p&gt;</comment>
                            <comment id="2175279" author="william.schultz" created="Thu, 7 Mar 2019 23:17:18 +0000"  >&lt;p&gt;After investigating the code to see if this bug is possible, here are some of my findings. My goal was to exercise the case where a change stream read executes in the middle of secondary batch application, after oplog entries &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5a7ccd5f073341f5f98e0ced39a80416869c181b/src/mongo/db/repl/sync_tail.cpp#L1345-L1346&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;have been written&lt;/a&gt; but before they have &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5a7ccd5f073341f5f98e0ced39a80416869c181b/src/mongo/db/repl/sync_tail.cpp#L1356&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;been applied&lt;/a&gt;. It was theorized that this would cause the query to potentially return a batch of oplog entries that haven&apos;t been applied yet, and so an update lookup query would read data earlier than the change events returned. It turns out that the oplog visibility point on secondaries is updated synchronously, in accordance with the completion of application of a batch. That is, when the oplog entries have been both written &lt;em&gt;and&lt;/em&gt; applied. This is done inside the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/5a7ccd5f073341f5f98e0ced39a80416869c181b/src/mongo/db/repl/sync_tail.cpp#L1379&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;replicationBatchIsComplete()&lt;/tt&gt;&lt;/a&gt; method, which will make a call to &lt;tt&gt;WiredTigerOplogManager::triggerJournalFlush()&lt;/tt&gt;, which signals the oplog visibility thread to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b21647a5e19e5980186e57f77be0b1e0ef92a087/src/mongo/db/storage/wiredtiger/wiredtiger_oplog_manager.cpp#L168-L169&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;wake up&lt;/a&gt; and advance the oplog read timestamp. The other place we trigger journal flushes like this is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/951a9eaba9bf9069866aa04e9a450d25cff74d1a/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L407-L415&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;inside&lt;/a&gt; &lt;tt&gt;WiredTigerRecoveryUnit::_txnClose()&lt;/tt&gt;, but we have special rules for when to actually trigger a journal flush there. The comment there indicates this:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// We only need to update oplog visibility where commits can be out-of-order with&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// respect to their assigned optime and such commits might otherwise be visible.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// This should happen only on primary nodes.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;_oplogManager-&amp;gt;triggerJournalFlush();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;So, this means that the oplog visibility point on secondaries should never advance in the middle of batch application. This implies that we will never read oplog entries past the newest timestamp of the most recent successful oplog batch, since change stream oplog scans are restricted from reading past the visibility timestamp. This should prevent the case of update lookup queries reading the pre-image of a returned change event. &lt;/p&gt;

&lt;p&gt;Since speculative majority change stream queries do not read at the lastApplied timestamp, though, update lookup queries may still be at risk of reading concurrently with secondary batch application. I may still need to investigate that issue. My understanding was that one of the reasons we do not allow reads during secondary batch application is because we may temporarily violate unique index constraints. I wonder if, since update lookup queries are always on a single document, they would ever be able to observe an inconsistent state. Even though writes are applied in parallel on secondaries, every document should presumably go through a series of states that is the same as it was on primary, even if the global database state is not consistent. It&apos;s still probably not a great idea to allow update lookup reads during batch application, though. &lt;/p&gt;</comment>
                            <comment id="2141463" author="william.schultz" created="Thu, 7 Feb 2019 18:53:59 +0000"  >&lt;p&gt;This is a hypothesized bug but has not been reproduced yet.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="510422">SERVER-33846</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="682368">SERVER-39356</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 8 Mar 2019 14:26:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 47 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1191</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 47 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hulbj3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hub61b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2784">Repl 2019-02-25</customfieldvalue>
    <customfieldvalue id="2822">Repl 2019-03-11</customfieldvalue>
    <customfieldvalue id="2823">Repl 2019-03-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hukxsf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>