<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:01:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42925] idempotency tests have oplog visiblity issues</title>
                <link>https://jira.mongodb.org/browse/SERVER-42925</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The MultiApplier reads from the oplog to piece together a multi-document prepared transaction.  The code is currently affected by oplog visibility issues, surfaced in an idempotency test.  This ticket is to fix those problems.&lt;/p&gt;


&lt;p&gt;Original description:&lt;br/&gt;
For example, this line:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/36aacf391a13eba67648f1cf625fe26515454a70/src/mongo/db/repl/sync_tail_test.cpp#L3072&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/36aacf391a13eba67648f1cf625fe26515454a70/src/mongo/db/repl/sync_tail_test.cpp#L3072&lt;/a&gt;&lt;br/&gt;
Attempts to process a prepare and a commit oplog entry in the same batch.  By tweaking timing, I can get that test to fail with this error:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-08-20T14:18:22.223-0400 F  -        [main] Fatal assertion 31145 IncompleteTransactionHistory: oplog no longer contains the complete write history of this transaction, log with opTime { ts: Timestamp(90, 0), t: 1 } cannot be found at src/mongo/db/transaction_history_iterator.cpp 137&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;There are several tests in sync_tail_test that attempt to do this.  It isn&apos;t clear to me why the test is currently passing most of the time.  I managed to trigger this failure simply by switching the underlying storage engine from emphemeralForTest to wiredTiger.&lt;/p&gt;</description>
                <environment></environment>
        <key id="901396">SERVER-42925</key>
            <summary>idempotency tests have oplog visiblity issues</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="lingzhi.deng@mongodb.com">Lingzhi Deng</assignee>
                                    <reporter username="milkie@mongodb.com">Eric Milkie</reporter>
                        <labels>
                    </labels>
                <created>Tue, 20 Aug 2019 18:21:20 +0000</created>
                <updated>Sun, 29 Oct 2023 22:17:53 +0000</updated>
                            <resolved>Fri, 4 Oct 2019 00:59:42 +0000</resolved>
                                                    <fixVersion>4.3.1</fixVersion>
                    <fixVersion>4.2.2</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="2559211" author="xgen-internal-githook" created="Thu, 21 Nov 2019 15:15:19 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;, &apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;username&apos;: &apos;ldennis&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42925&quot; title=&quot;idempotency tests have oplog visiblity issues&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42925&quot;&gt;&lt;del&gt;SERVER-42925&lt;/del&gt;&lt;/a&gt; Refactor idempotency tests oplog application and fix oplog visiblity issues&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/5e1b0d1a69be18eb4f93f01831d9fd9db1b3a516&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/5e1b0d1a69be18eb4f93f01831d9fd9db1b3a516&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2465690" author="xgen-internal-githook" created="Fri, 4 Oct 2019 00:55:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Lingzhi Deng&apos;, &apos;username&apos;: &apos;ldennis&apos;, &apos;email&apos;: &apos;lingzhi.deng@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42925&quot; title=&quot;idempotency tests have oplog visiblity issues&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42925&quot;&gt;&lt;del&gt;SERVER-42925&lt;/del&gt;&lt;/a&gt; Refactor idempotency tests oplog application and fix oplog visiblity issues&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d1c6b36bf04dd90ad963ad5dfbfa491de0c88789&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d1c6b36bf04dd90ad963ad5dfbfa491de0c88789&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2444113" author="lingzhi.deng" created="Tue, 1 Oct 2019 21:22:21 +0000"  >&lt;p&gt;After talking to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, I think there is no issue in the production batch application since we always &lt;a href=&quot;https://github.com/mongodb/mongo/blob/12011588f517f910ea5b6a34591755b3ca04223d/src/mongo/db/repl/oplog_applier_impl.cpp#L279-L282&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;update oplog visibility after each batch&lt;/a&gt;. In other words, we don&apos;t rely on  So the fix would be to fix idempotency tests to simulate secondary&apos;s behavior.&lt;/p&gt;</comment>
                            <comment id="2430996" author="lingzhi.deng" created="Tue, 24 Sep 2019 15:40:31 +0000"  >&lt;p&gt;Yes, I think it would be helpful to have a way to ignore the visibility rules. I was about to open a CR for this which introduces a new read source to skip setting the &lt;tt&gt;_oplogVisibleTs&lt;/tt&gt;. Do you think this is the way to go (kinda hacky?) or do you think it is better to do in the cursor layer as described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43412&quot; title=&quot;Create a wrapper around getCursor calls for the oplog collection that ensures oplog visibility rules are enforced&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43412&quot;&gt;&lt;del&gt;SERVER-43412&lt;/del&gt;&lt;/a&gt;? Is the plan to provide a flag in getCursor so Query code path can set/unset it?&lt;/p&gt;</comment>
                            <comment id="2430504" author="dianna.hohensee" created="Tue, 24 Sep 2019 13:44:50 +0000"  >&lt;p&gt;Hi all, Dan.G pointed me at this tickets existence. I just filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-43412&quot; title=&quot;Create a wrapper around getCursor calls for the oplog collection that ensures oplog visibility rules are enforced&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-43412&quot;&gt;&lt;del&gt;SERVER-43412&lt;/del&gt;&lt;/a&gt;, which still needs to be explored some for viability, but seems like it would help address this issue by providing a code path to ignore oplog visibility rules?&lt;/p&gt;</comment>
                            <comment id="2391690" author="milkie" created="Tue, 27 Aug 2019 22:27:08 +0000"  >&lt;p&gt;Now that MMAP is gone, we can probably change the query and storage systems to use OplogHack for both forward and reverse cursors.&lt;br/&gt;
We could also consider adding a query flag that tells the query system to simply ignore oplog visibility even for forward cursors, for internal use, but implementing that would be a lot of work.&lt;/p&gt;</comment>
                            <comment id="2391497" author="siyuan.zhou@10gen.com" created="Tue, 27 Aug 2019 21:29:01 +0000"  >&lt;p&gt;TransactionHistoryIterator is also needed by change stream so it may have to use OplogHack for performance reasons, which is only for forward cursors.&lt;/p&gt;</comment>
                            <comment id="2389988" author="milkie" created="Tue, 27 Aug 2019 18:13:22 +0000"  >&lt;p&gt;Yes, those symptoms helped me discover the problem.  It was very subtle, and it is a race, which is why when you add things to slow down the test slightly (like logging), it changes the behavior.&lt;/p&gt;

&lt;p&gt;The issue is that the code as written is not waiting for the oplog visibility thread to make the oplog entries visible.  Typically, doing a query on the oplog does do this wait for you automatically, but only on primary nodes as per the comment here:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/1fa4766c621bd4cfd74319094469eff3a5de3b79/src/mongo/db/query/get_executor.cpp#L299&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/1fa4766c621bd4cfd74319094469eff3a5de3b79/src/mongo/db/query/get_executor.cpp#L299&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I think this is actually identifying a problem even in production, although the code in production does so much in between writing a batch to the oplog and processing the next batch that I guess we never encounter it.  The way to fix it would be to call waitForAllEarlierOplogWritesToBeVisible() at the end of every batch, although this would needlessly affect performance.  A better solution would be to use a reverse-scanning cursor when doing the oplog entry queries in the TransactionHistoryIterator.  Unlike forward cursors, reverse cursors skip oplog visiblity rules and can see everything immediately.&lt;/p&gt;</comment>
                            <comment id="2388733" author="lingzhi.deng" created="Tue, 27 Aug 2019 15:23:30 +0000"  >&lt;p&gt;Before it calls &lt;tt&gt;fillWriterVectors()&lt;/tt&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/1fa4766c621bd4cfd74319094469eff3a5de3b79/src/mongo/db/repl/idempotency_test_fixture.cpp#L375-L382&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;it writes all oplog entries to the oplog collection first&lt;/a&gt;. I was able to reproduce the issue and it seems that this only happens with the &lt;tt&gt;CommitPreparedTransactionIgnoresNamespaceNotFoundErrors&lt;/tt&gt; test. I am not sure why isn&apos;t the prepare oplog entry (inserted by &lt;tt&gt;getStorageInterface()&amp;#45;&amp;gt;insertDocument&lt;/tt&gt;) visible to the &lt;tt&gt;TransactionHistoryIterator&lt;/tt&gt;. I tried to do a &lt;tt&gt;DBDirectClient&lt;/tt&gt; find after the &lt;tt&gt;insertDocument&lt;/tt&gt; call and I was able to see the prepare oplog entry. And surprisingly the problem goes away if I do a find first before calling &lt;tt&gt;fillWriterVectors()&lt;/tt&gt;. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, do you see anything obviously wrong with the use of &lt;tt&gt;getStorageInterface()&amp;#45;&amp;gt;insertDocument&lt;/tt&gt;?&lt;/p&gt;</comment>
                            <comment id="2387118" author="milkie" created="Mon, 26 Aug 2019 19:05:44 +0000"  >&lt;p&gt;Before the code gets to runOpPtrsInitialSync, though, it calls fillWriterVectors() on each of the ops that was passed to testOpsAreIdempotent().  It is fillWriterVectors() that, when encountering a commit op, does the search in the oplog for the associated oplog entries.  This is the problem.&lt;/p&gt;</comment>
                            <comment id="2387060" author="lingzhi.deng" created="Mon, 26 Aug 2019 18:37:58 +0000"  >&lt;p&gt;I don&apos;t think idempotency tests prepare and commit in the same batch. &lt;tt&gt;testOpsAreIdempotent&lt;/tt&gt; calls &lt;a href=&quot;https://github.com/mongodb/mongo/blob/1fa4766c621bd4cfd74319094469eff3a5de3b79/src/mongo/db/repl/sync_tail_test_fixture.cpp#L260-L270&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;runOpPtrsInitialSync&lt;/tt&gt; which runs &lt;tt&gt;multiSyncApply&lt;/tt&gt; to apply each operation in its own batch&lt;/a&gt;. &lt;/p&gt;</comment>
                            <comment id="2381391" author="milkie" created="Wed, 21 Aug 2019 18:12:33 +0000"  >&lt;p&gt;I don&apos;t think that commit would fix this problem; I think the test code is still wrong.  I am using the head of master branch and by changing the idempotency_test_fixture to use WiredTiger, the unit tests fail 100% with the error seen in the description, so I think it&apos;s a still a problem.&lt;/p&gt;</comment>
                            <comment id="2381102" author="judah.schvimer" created="Wed, 21 Aug 2019 15:43:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, do we know if this got fixed in &lt;a href=&quot;https://github.com/mongodb/mongo/commit/f14d9c4c07e69c2109de0af059123060c73cdd77#diff-cc663e08c6c92a191f0d876b013d382b?&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f14d9c4c07e69c2109de0af059123060c73cdd77#diff-cc663e08c6c92a191f0d876b013d382b?&lt;/a&gt; The BF seems to have stopped then.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="855724">SERVER-42213</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 21 Aug 2019 15:43:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 11 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 11 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dianna.hohensee@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvlwt3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvaq13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3201">Repl 2019-09-09</customfieldvalue>
    <customfieldvalue id="3202">Repl 2019-09-23</customfieldvalue>
    <customfieldvalue id="3260">Repl 2019-10-07</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvlj2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>