<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:37:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34683] Downgrade replicaset from 3.6.4 to 3.4.14 fails due to the presence of `config.system.sessions`</title>
                <link>https://jira.mongodb.org/browse/SERVER-34683</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi&lt;/p&gt;

&lt;p&gt;I made an update of my RS from 3.4.1 to 3.6.4,&lt;br/&gt;
but application where not using newest driver yet,&lt;br/&gt;
so I was enforced to downgrade back&lt;br/&gt;
(acc to docu, I have chosen the newest 3.4.14 ).&lt;/p&gt;

&lt;p&gt;Downgrade was not possible. Even with data removal (enforce &quot;scratch-replication&quot;).&lt;br/&gt;
I constantly got error, about config.system.session collection :&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;018-04-25T17:21:01.319+0200 F REPL     [repl writer worker 1] writer worker caught exception: 10156 cannot update system collection: config.system.sessions q: { _id: { id: BinData(4, 96E23194367D482B81B1D75B6423AFBD), uid: BinData(0, EEDC58CB4933CBA895757E2FD8275E869A732F964407D28FA4D462EE4FC5B046) } } u: { $set: { lastUse: new Date(1524669659199) } } on: { ts: Timestamp 1524669659000|10, t: 39, h: 6122200662662238595, v: 2, op: &quot;u&quot;, ns: &quot;config.system.sessions&quot;, o2: { _id: { id: BinData(4, 96E23194367D482B81B1D75B6423AFBD), uid: BinData(0, EEDC58CB4933CBA895757E2FD8275E869A732F964407D28FA4D462EE4FC5B046) } }, wall: new Date(1524669659197), o: { $set: { lastUse: new Date(1524669659199) } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-04-25T17:21:01.319+0200 I -        [repl writer worker 1] Fatal assertion 16359 Location10156: cannot update system collection: config.system.sessions q: { _id: { id: BinData(4, 96E23194367D482B81B1D75B6423AFBD), uid: BinData(0, EEDC58CB4933CBA895757E2FD8275E869A732F964407D28FA4D462EE4FC5B046) } } u: { $set: { lastUse: new Date(1524669659199) } } at src/mongo/db/repl/sync_tail.cpp 1082&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-04-25T17:21:01.319+0200 F REPL     [repl writer worker 15] writer worker caught exception: 10156 cannot update system collection: config.system.sessions q: { _id: { id: BinData(4, F7749D80D99044C18AEC4F16DB0841A6), uid: BinData(0, 575CD72AF2368BDED794D83BD07BA14894021281F75D97BE815F1FE5D795D1E1) } } u: { $set: { lastUse: new Date(1524669659199) } } on: { ts: Timestamp 1524669659000|6, t: 39, h: 9214092136229990651, v: 2, op: &quot;u&quot;, ns: &quot;config.system.sessions&quot;, o2: { _id: { id: BinData(4, F7749D80D99044C18AEC4F16DB0841A6), uid: BinData(0, 575CD72AF2368BDED794D83BD07BA14894021281F75D97BE815F1FE5D795D1E1) } }, wall: new Date(1524669659197), o: { $set: { lastUse: new Date(1524669659199) } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-04-25T17:21:01.319+0200 I -        [repl writer worker 1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after fassert() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;


&lt;p&gt;I have even tried to drop the collection and db config&lt;br/&gt;
but it was recreated.&lt;/p&gt;</description>
                <environment></environment>
        <key id="534419">SERVER-34683</key>
            <summary>Downgrade replicaset from 3.6.4 to 3.4.14 fails due to the presence of `config.system.sessions`</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="misha.tyulenev@mongodb.com">Misha Tyulenev</assignee>
                                    <reporter username="sielaq">Wojciech Sielski</reporter>
                        <labels>
                    </labels>
                <created>Thu, 26 Apr 2018 06:41:32 +0000</created>
                <updated>Sun, 29 Oct 2023 22:32:22 +0000</updated>
                            <resolved>Thu, 10 May 2018 00:58:27 +0000</resolved>
                                    <version>3.6.4</version>
                                    <fixVersion>3.6.5</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="1930822" author="ramon.fernandez" created="Mon, 25 Jun 2018 17:47:06 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=victorgp&quot; class=&quot;user-hover&quot; rel=&quot;victorgp&quot;&gt;victorgp&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=robomon1&quot; class=&quot;user-hover&quot; rel=&quot;robomon1&quot;&gt;robomon1&lt;/a&gt;, since this ticket corresponds to a specific bug and it&apos;s been closed already I&apos;d request that you either post on &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; if you have a support-related question, or open a new SERVER ticket if you believe you&apos;ve found a bug.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1928285" author="robomon1" created="Thu, 21 Jun 2018 21:55:32 +0000"  >&lt;p&gt;Not sure this is totally fixed in 3.6.5.&#160; I just did a rolling upgrade of a 3 node replicaset that was on 3.4 with FCV=3.4.&#160; The nodes were aws instances and it was easy enough to just wipe them out and recreate them with 3.6.5.&#160; Nodes 1 and 2 went fine.&#160; When I ran the rs.stepDown() on the primary it took more than a few seconds to select a new primary.&#160; Then the mongod service on Node 3 which was still 3.4 aborted with this&#160; error.&#160; After that I couldn&apos;t restart the service on Node 3.&#160; I finally just upgraded Node 3 to 3.6 and everything started fine.&lt;/p&gt;</comment>
                            <comment id="1925182" author="victorgp" created="Tue, 19 Jun 2018 17:25:25 +0000"  >&lt;p&gt;I&apos;m affected by this issue in a slightly different way.&lt;/p&gt;

&lt;p&gt;We downgraded fom 3.6 to 3.4 and in one of the shards of the cluster the config.system.sessions collection remained there, not in the rest. It is not in the config servers either, so what i&apos;m trying to do is drop the collection it in that shard, but i cannot find a role or set of roles that allow me to do that.&lt;/p&gt;

&lt;p&gt;I tried the admin, restore and root roles with no luck, i always get: &quot;not authorized on config to execute command&quot;&lt;/p&gt;

&lt;p&gt;Do you know what permissions i need to set to perform this operation in that shard?&lt;/p&gt;</comment>
                            <comment id="1887710" author="xgen-internal-githook" created="Thu, 10 May 2018 00:56:12 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Misha Tyulenev&apos;, &apos;email&apos;: &apos;misha@mongodb.com&apos;, &apos;username&apos;: &apos;mikety&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34683&quot; title=&quot;Downgrade replicaset from 3.6.4 to 3.4.14 fails due to the presence of `config.system.sessions`&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34683&quot;&gt;&lt;del&gt;SERVER-34683&lt;/del&gt;&lt;/a&gt; drop config.system.sessions on downgrade to 3.4&lt;br/&gt;
Branch: v3.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8d736eabdbc0da2d4846edffad26df538a6adad4&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8d736eabdbc0da2d4846edffad26df538a6adad4&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1881502" author="kevin.pulo@10gen.com" created="Thu, 3 May 2018 12:48:18 +0000"  >&lt;p&gt;For users currently affected by this issue (running 3.6.4, with FCV 3.4 (either never set to 3.6, or set back to 3.4), attempting to downgrade to 3.4, and encountering these errors on the 3.4 nodes), another potential workaround is to rolling restart on 3.6.4 with &lt;tt&gt;--setParameter disableLogicalSessionCacheRefresh=true&lt;/tt&gt;, and then perform the rolling downgrade to 3.4 while also removing this parameter when starting each 3.4 mongod (since it will prevent 3.4 mongod from starting).&lt;/p&gt;

&lt;p&gt;Note that this is an &lt;b&gt;undocumented internal-only parameter that should not be used otherwise&lt;/b&gt; &amp;#8212; while set to true it will inhibit the writes to &lt;tt&gt;config.system.sessions&lt;/tt&gt; &amp;#8212; which are normal and necessary in homogenous 3.6 replica sets, but the source of this issue in mixed 3.4/3.6 replica sets.&lt;/p&gt;</comment>
                            <comment id="1878943" author="schwerin" created="Tue, 1 May 2018 12:57:04 +0000"  >&lt;p&gt;This plan sounds reasonable. We should also see about improving the downgrade test coverage in the multiversion suite.&lt;/p&gt;</comment>
                            <comment id="1878416" author="renctan" created="Mon, 30 Apr 2018 20:44:13 +0000"  >&lt;p&gt;The approach sounds reasonable to me&lt;/p&gt;</comment>
                            <comment id="1878353" author="kaloian.manassiev" created="Mon, 30 Apr 2018 20:01:47 +0000"  >&lt;p&gt;Currently (as of 3.6.4) the &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection is unconditionally &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.4/src/mongo/db/sessions_collection_config_server.cpp#L101&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;created by the config server&lt;/a&gt; and all other nodes (shards and mongos) just check for its presence before attempting to write to it.&lt;/p&gt;

&lt;p&gt;In order to fix this downgrade problem, I propose that we make the following changes:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Change &lt;tt&gt;SessionsCollectionConfigServer&lt;/tt&gt; to only create the collection if FCV is 3.6 (this will require it taking the FCV lock as well in order to serialize with possible concurrent FCV changes)&lt;/li&gt;
	&lt;li&gt;On FCV downgrade to 3.4, drop the sharded &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection, driven by the config server again.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The last step still has a race condition where a stray write to the &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection may accidentally recreate it on the config server, so I propose to also disallow the creation of &lt;tt&gt;config.system.sessions&lt;/tt&gt; at all if FCV is not 3.6.&lt;/p&gt;

&lt;p&gt;This is the general direction and some race conditions may still have to be fleshed out, but I wanted to verify that the direction sounds correct before we put more time into designing it. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;With these fixes, customers who are on 3.6.4 and are unable to downgrade to 3.4 will have two options:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Manually delete the &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection on all nodes (config server and shards)&lt;/li&gt;
	&lt;li&gt;Upgrade to 3.6.5, set FCV to 3.4 and then downgrade to 3.4&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1874785" author="kaloian.manassiev" created="Thu, 26 Apr 2018 12:31:52 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sielaq&quot; class=&quot;user-hover&quot; rel=&quot;sielaq&quot;&gt;sielaq&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed report and for confirming that the feature compatibility version has been downgraded to 3.4.&lt;/p&gt;

&lt;p&gt;From cursory look it appears that the code which downgrades the FCV omits dropping the internal &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection (which is something new we introduced in 3.6 in order to support logical sessions) or it somehow gets recreated after the FCV downgrade (more likely).&lt;/p&gt;

&lt;p&gt;While we investigate this issue further, if you are blocked because of the failing downgrade, you can work around the problem by manually dropping the &lt;tt&gt;config.system.sessions&lt;/tt&gt; collection.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1874675" author="sielaq" created="Thu, 26 Apr 2018 07:07:52 +0000"  >&lt;p&gt;overtaking the coming question: yes downgrading features to 3.4 has been done - all acc to procedure&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000aQv1aIAC]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 26 Apr 2018 12:31:52 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 33 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 33 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>kevin.pulo@mongodb.com</customfieldvalue>
            <customfieldvalue>misha.tyulenev@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
            <customfieldvalue>robomon1</customfieldvalue>
            <customfieldvalue>victorgp</customfieldvalue>
            <customfieldvalue>sielaq</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htwhmv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8ztb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2257">Sharding 2018-05-07</customfieldvalue>
    <customfieldvalue id="2290">Sharding 2018-05-21</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;3.4 upgrade to 3.6.4 and try to downgrade, when config.system.session exists&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htw3v3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>