<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:27:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31691] Change streams UUID mismatch uasserts with error code that causes invariant</title>
                <link>https://jira.mongodb.org/browse/SERVER-31691</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Found while writing the tests for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31392&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;SERVER-31392&lt;/a&gt;, the DocumentSourceChangeStream hits a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/pipeline_d.cpp#L366&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;uassert&lt;/a&gt; when the UUID in the oplog entry mismatches the UUID in the scoped collection metadata.  Returning ErrorCodes::StaleConfig results in a dynamic_cast a few layers up, and eventually an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/service_entry_point_mongod.cpp#L722&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;invariant&lt;/a&gt;.  I am not sure we should hit this uassert at all (a separate discussion), but for sure it doesn&apos;t seem like we should be returning this status.&lt;/p&gt;</description>
                <environment></environment>
        <key id="449940">SERVER-31691</key>
            <summary>Change streams UUID mismatch uasserts with error code that causes invariant</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="bernard.gorman@mongodb.com">Bernard Gorman</assignee>
                                    <reporter username="nicholas.zolnierz@mongodb.com">Nicholas Zolnierz</reporter>
                        <labels>
                    </labels>
                <created>Tue, 24 Oct 2017 14:10:48 +0000</created>
                <updated>Mon, 30 Oct 2023 23:12:33 +0000</updated>
                            <resolved>Tue, 7 Nov 2017 23:40:32 +0000</resolved>
                                                    <fixVersion>3.6.0-rc4</fixVersion>
                                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1953962" author="sjohnson540" created="Mon, 23 Jul 2018 17:35:13 +0000"  >&lt;p&gt;Awesome. that fixed the problem. Thank you.&lt;/p&gt;</comment>
                            <comment id="1953254" author="bernard.gorman" created="Sun, 22 Jul 2018 22:54:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sjohnson540&quot; class=&quot;user-hover&quot; rel=&quot;sjohnson540&quot;&gt;sjohnson540&lt;/a&gt;, many thanks for providing that additional information. After conferring with some of my colleagues, we believe that you are likely encountering the issue described in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36154&quot; title=&quot;Shard&amp;#39;s in-memory CSS is not refreshed after upgrading from 3.4 to 3.6, causing a UUID mismatch on $changeStream operations&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36154&quot;&gt;&lt;del&gt;SERVER-36154&lt;/del&gt;&lt;/a&gt;. MongoDB 3.6 introduced &lt;a href=&quot;https://docs.mongodb.com/manual/core/databases-and-collections/#unique-identifiers&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;per-collection UUIDs&lt;/a&gt;, and when upgrading from 3.4 to 3.6 every existing collection is assigned a new UUID &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/3.6-upgrade-sharded-cluster/#enable-backwards-incompatible-newversion-features&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;when the &lt;tt&gt;featureCompatibilityVersion&lt;/tt&gt; is set to 3.6&lt;/a&gt;. However, while the UUIDs for each collection will be persisted to disk, it is possible that some nodes may fail to update their &lt;b&gt;in-memory cache of the collection state&lt;/b&gt; to&#160;reflect this. When a &lt;tt&gt;$changeStream&lt;/tt&gt; is opened on a collection whose cache does not contain the new UUID, it results in the error you described above.&lt;/p&gt;

&lt;p&gt;To resolve the issue, please try the following:&lt;/p&gt;

&lt;p&gt;1. Connect directly to each shard&apos;s Primary, and issue the command below to refresh the Primary&apos;s in-memory collection state cache:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({forceRoutingTableRefresh: &quot;databaseName.collectionName&quot;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;2. Unfortunately, this will only refresh the Primary&apos;s cache, and &lt;tt&gt;forceRoutingTableRefresh&lt;/tt&gt; can only be run on the Primary. To refresh the Secondaries&apos; caches, please restart each Secondary one by one. This should fully resolve the problem without causing any shard downtime.&lt;/p&gt;

&lt;p&gt;Once this is done, you should not encounter the error again at any point in the future, since all &lt;tt&gt;mongoDs&lt;/tt&gt; will have an up-to-date cache including the UUIDs generated for each collection during the upgrade, and UUIDs will automatically be cached for any future collections created in 3.6 &lt;tt&gt;featureCompatibilityVersion&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Bernard&lt;/p&gt;</comment>
                            <comment id="1948412" author="sjohnson540" created="Mon, 16 Jul 2018 18:58:56 +0000"  >&lt;p&gt;@bernard.gorman I am not Basit, but I have run into what seems to be the exact same issue, I can answer the questions:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Yes, I am seeing this everytime I open a changestream&lt;/li&gt;
	&lt;li&gt;I never receive successful notifications&lt;/li&gt;
	&lt;li&gt;It has NOT been dropped/recreated&lt;/li&gt;
	&lt;li&gt;The shard key is the id&lt;/li&gt;
	&lt;li&gt;Output: (Mongos and the primary returns the same output)&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[[ { &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;name&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;actions&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;type&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;collection&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;options&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : { &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;flags&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; }, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;info&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : { &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;readOnly&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;false&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;uuid&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : UUID(&lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;4bfddfea-811e-4a59-935d-0f2fce3fdc0d&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) }, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;idIndex&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : { &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;v&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;key&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : { &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;_id&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; }, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;name&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;_id_&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;ns&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;staging.actions&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; } }]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I see this issue is closed. I am on version 3.6.5 and still encountering this issue.&lt;/p&gt;</comment>
                            <comment id="1857531" author="bernard.gorman" created="Fri, 6 Apr 2018 14:00:56 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=basitmustafa&quot; class=&quot;user-hover&quot; rel=&quot;basitmustafa&quot;&gt;basitmustafa&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;Thank you for bringing this to our attention. You&apos;re correct that the fix for this issue was included in the GA release of 3.6.0, but that fix related only to the &lt;b&gt;behaviour&lt;/b&gt; of the server when this scenario was encountered. Before this ticket, a UUID mismatch resulted in an invariant failure, which always causes the server to shut down as a protective measure. The fix implemented here was to change this to an assertion, which simply aborts the current operation and returns an error to the client.&lt;/p&gt;

&lt;p&gt;I&apos;m surprised to hear that you&apos;re seeing this exception on &lt;b&gt;every change&lt;/b&gt; from one particular collection. The assertion &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3.6.3/src/mongo/db/pipeline/pipeline_d.cpp#L835-L838&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;is thrown here&lt;/a&gt; in 3.6.3, in the &lt;tt&gt;collectionDocumentKeyFields&lt;/tt&gt; method. This is invoked &lt;a href=&quot;https://github.com/mongodb/mongo/blob/3.6.3/src/mongo/db/pipeline/document_source_change_stream.cpp#L428-L451&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the first time we encounter a change&lt;/a&gt; on the target collection; having extracted the UUID from the change&apos;s oplog entry, we then consult our cached collection metadata to determine whether the collection is sharded and, if so, what the shard key is.&lt;/p&gt;

&lt;p&gt;The only scenario in which I&apos;ve been able to force our tests to hit this assertion is if (1) a &lt;tt&gt;$changeStream&lt;/tt&gt; is opened on a collection, (2) some writes are performed but no changes have yet been retrieved by the stream, (3) the collection is dropped, recreated (causing it to acquire a new UUID) and sharded, and (4) a &lt;tt&gt;getMore&lt;/tt&gt; is issued on the &lt;tt&gt;$changeStream&lt;/tt&gt;. This pulls the first of the changes, which still has the old UUID, through the pipeline; we attempt to look up the shard key, realise that the UUIDs no longer match, and throw the exception.&lt;/p&gt;

&lt;p&gt;However, it seems unlikely that such an artificial sequence of events is happening in your cluster. Can you let me know the following:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Are you seeing this every time you try to open a new &lt;tt&gt;$changeStream&lt;/tt&gt;? Or when you try to resume a stream?&lt;/li&gt;
	&lt;li&gt;Do you ever receive &lt;b&gt;any&lt;/b&gt; successful notifications before the error?&lt;/li&gt;
	&lt;li&gt;Has this collection been dropped and recreated at any point?&lt;/li&gt;
	&lt;li&gt;What&apos;s the collection&apos;s shard key?&lt;/li&gt;
	&lt;li&gt;Is this collection distributed across all shards in the cluster, or zoned to a subset of shards?&lt;/li&gt;
	&lt;li&gt;What is the output of running the following commands on &lt;tt&gt;mongos&lt;/tt&gt; (in particular, what is the value of the &lt;tt&gt;info.uuid&lt;/tt&gt; field)?
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;use vtDataWarehouse&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.getCollectionInfos({name: &quot;VTDataObject&quot;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;If you connect directly to the Primary of each shard and run the same commands as above, do the UUIDs on all of them match the UUID you observed via &lt;tt&gt;mongoS&lt;/tt&gt;?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Best regards,&lt;br/&gt;
Bernard&lt;/p&gt;</comment>
                            <comment id="1847077" author="JIRAUSER1262896" created="Tue, 27 Mar 2018 20:26:53 +0000"  >&lt;p&gt;I think we are seeing this behavior in the wild in a production release (although I am not intimately familiar with the code base, we are simply a production user, not yet contributors).&lt;/p&gt;

&lt;p&gt;On a sharded collection (vtDataWarehouse.VTDataObject) any change events result in an error, but only on this specific collection. Other collections, both sharded and unsharded, things work fine on the same system, same connection, even.&lt;/p&gt;

&lt;p&gt;Any advice on a workaround before the supposed 3.7 change around the uassert behavior (although this was supposed fixed for 3.6 if I am understanding correctly)? &lt;/p&gt;

&lt;p&gt;error:  MongoError: Collection vtDataWarehouse.VTDataObject UUID differs from UUID on change stream operations&lt;br/&gt;
    at /Users/basitmustafa/GitHub/voltaire-api-v5/node_modules/mongodb-core/lib/connection/pool.js:595:61&lt;br/&gt;
    at authenticateStragglers (/Users/basitmustafa/GitHub/voltaire-api-v5/node_modules/mongodb-core/lib/connection/pool.js:513:16)&lt;br/&gt;
    at Connection.messageHandler (/Users/basitmustafa/GitHub/voltaire-api-v5/node_modules/mongodb-core/lib/connection/pool.js:549:5)&lt;br/&gt;
    at emitMessageHandler (/Users/basitmustafa/GitHub/voltaire-api-v5/node_modules/mongodb-core/lib/connection/connection.js:309:10)&lt;br/&gt;
    at Socket.&amp;lt;anonymous&amp;gt; (/Users/basitmustafa/GitHub/voltaire-api-v5/node_modules/mongodb-core/lib/connection/connection.js:452:17)&lt;br/&gt;
    at Socket.emit (events.js:180:13)&lt;br/&gt;
    at addChunk (_stream_readable.js:269:12)&lt;br/&gt;
    at readableAddChunk (_stream_readable.js:256:11)&lt;br/&gt;
    at Socket.Readable.push (_stream_readable.js:213:10)&lt;br/&gt;
    at TCP.onread (net.js:578:20)&lt;/p&gt;

&lt;p&gt;Server is 3.6.3 gitVersion 9586e557d54ef70f9ca4b43c26892cd55257e1a5 on ubuntu 1604 x86_64&lt;/p&gt;</comment>
                            <comment id="1719956" author="bernard.gorman" created="Tue, 7 Nov 2017 23:58:22 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt;: having looked into this, I&apos;m not convinced that &lt;tt&gt;uasserting&lt;/tt&gt; here is correct or desirable behaviour. I&apos;ve pushed a fix to avoid the invariant failure and have opened &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31847&quot; title=&quot;Allow sharded $changeStream to continue if collection is dropped and resharded before documentKey is obtained&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31847&quot;&gt;&lt;del&gt;SERVER-31847&lt;/del&gt;&lt;/a&gt; to track followup work.&lt;/p&gt;</comment>
                            <comment id="1719940" author="xgen-internal-githook" created="Tue, 7 Nov 2017 23:39:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Bernard Gorman&apos;, &apos;username&apos;: &apos;gormanb&apos;, &apos;email&apos;: &apos;bernard.gorman@gmail.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31691&quot; title=&quot;Change streams UUID mismatch uasserts with error code that causes invariant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31691&quot;&gt;&lt;del&gt;SERVER-31691&lt;/del&gt;&lt;/a&gt; Change streams UUID mismatch uasserts with error code that causes invariant&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9d82100f7a0a4df9e797eef8f0fb3f6c66938771&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9d82100f7a0a4df9e797eef8f0fb3f6c66938771&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1711335" author="spencer" created="Fri, 27 Oct 2017 17:14:43 +0000"  >&lt;p&gt;At this point I think we shouldn&apos;t worry about trying to properly trigger a metadata refresh.  The only case it can possibly be an issue is after an unsharded collection becomes sharded, which has other known issues right now anyway, and it&apos;s very easy to work around the issue (just do any versioned read or write against the shard, which will kick off a metadata refresh).  So I think just using a normal uassert code should be fine for now.&lt;/p&gt;</comment>
                            <comment id="1711327" author="spencer" created="Fri, 27 Oct 2017 17:08:41 +0000"  >&lt;p&gt;Yes, that is a known issue and should be fine.  If you haven&apos;t even returned any results yet then there should really be no problem in just re-establishing a brand new cursor.  So maybe just the test you were working on just needs to be updated to make sure it has processed at least one result before dropping the collection?  Although we also need to stop the uassert from triggering an invariant, but I think we&apos;re usasserting at the right time at least.&lt;/p&gt;</comment>
                            <comment id="1711226" author="bernard.gorman" created="Fri, 27 Oct 2017 16:12:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt;, I think that makes sense. From the code, we do check &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/document_source_change_stream.cpp#L388&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;whether we&apos;ve already obtained and stored the shard key fields in &lt;tt&gt;_documentKeyFields&lt;/tt&gt;&lt;/a&gt; before consulting the metadata.&lt;/p&gt;

&lt;p&gt;However, that check &lt;b&gt;only happens the first time we pull documents through the transform stage&lt;/b&gt;. So if the UUID changes between the time that the cursor is established on the shard and the point where it begins returning results, then I can see how we would hit this uassert.&lt;/p&gt;</comment>
                            <comment id="1711188" author="spencer" created="Fri, 27 Oct 2017 15:56:07 +0000"  >&lt;p&gt;I think using the StaleConfig code is wrong, and we probably don&apos;t have the right information available to throw StaleConfigException properly.&lt;/p&gt;

&lt;p&gt;What I want to understand is if we&apos;ve already saved the shard key onto the cursor and checked the UUID, why are we hitting this uassert at all on the getMore?  The idea behind how we extract the shard key for insert entries was that for a sharded changestream, when the mongod processes its very first notification it will check the collection metadata, extract the shard key, confirm the UUID of the collection metadata matches the UUID of the notification, then store the shard key on the cursor.  From that point on, for the lifetime of the cursor, the mongod shouldn&apos;t have to be checking the CollectionMetadata at all since the cursor already knows the shard key, so even if the collection is dropped it should be able to continue processing entries until the cursor catches up to the drop entry and returns an &quot;invalidate&quot;.&lt;/p&gt;

&lt;p&gt;The only case that should fail is if the cursor is lost and the stream has to be resumed with a ResumeToken, making a new cursor.  The new cursor will try to load the CollectionMetadata but won&apos;t find any information for the collection with that name.  In that case we should return a sane error back to the user.  It is a known issue that we can&apos;t resume after a collection drop, but so long as we&apos;re using the same cursor it should work fine.&lt;/p&gt;</comment>
                            <comment id="1711071" author="bernard.gorman" created="Fri, 27 Oct 2017 14:31:03 +0000"  >&lt;p&gt;See dupe ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31746&quot; title=&quot;Invariant failure on sharded $changeStream UUID mismatch&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31746&quot;&gt;&lt;del&gt;SERVER-31746&lt;/del&gt;&lt;/a&gt; for a slightly different manifestation of this issue. If we open a &lt;tt&gt;$changeStream&lt;/tt&gt; on a sharded collection and that collection is dropped before we have iterated all oplog entries, this &lt;tt&gt;uassert&lt;/tt&gt; prevents us from ever seeing those changes.&lt;/p&gt;</comment>
                            <comment id="1707684" author="nicholas.zolnierz" created="Tue, 24 Oct 2017 18:23:16 +0000"  >&lt;p&gt;As far as I can tell, the collection metadata&apos;s UUID is valid at the time of issuing the getNext(), which would be after the collection is sharded.  If I create a new change stream cursor after sharding the collection, everything works fine and the collection&apos;s UUID is unchanged.&lt;/p&gt;

&lt;p&gt;The issue seems to be that the UUID associated with the change stream cursor created before sharding the collection is incorrect/stale/!missing.  At a first glance, perhaps this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/document_source_change_stream.cpp#L385&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;check&lt;/a&gt; was meant to catch it? &lt;/p&gt;</comment>
                            <comment id="1707639" author="spencer" created="Tue, 24 Oct 2017 17:56:18 +0000"  >&lt;p&gt;I has assumed that it was due to there being &lt;b&gt;no&lt;/b&gt; UUID in the CollectionMetadata since the config servers don&apos;t track any information about unsharded collections, but I&apos;ll leave &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nicholas.zolnierz&quot; class=&quot;user-hover&quot; rel=&quot;nicholas.zolnierz&quot;&gt;nicholas.zolnierz&lt;/a&gt; to confirm. &lt;/p&gt;</comment>
                            <comment id="1707626" author="esha.maharishi@10gen.com" created="Tue, 24 Oct 2017 17:48:31 +0000"  >&lt;p&gt;I&apos;m confused why Nick&apos;s case causes a UUID mismatch. The UUID shouldn&apos;t change when the collection becomes sharded? That sounds like a more serious issue...&lt;/p&gt;</comment>
                            <comment id="1707500" author="spencer" created="Tue, 24 Oct 2017 16:32:54 +0000"  >&lt;p&gt;Actually, on further consideration this may not actually be as big a problem in practice as I thought, since I don&apos;t think we look for the CollectionMetadata until we process the first actual changeStream notification.  By the time there is any actual notifications to return, there must have been at least 1 insert to the collection.  Since inserts are versioned, that must have caused the shard to refresh its CollectionMetadata at that point.&lt;/p&gt;

&lt;p&gt;So that means having a UUID mismatch between a real collection UUID in the CollectionMetadata and the UUID in the oplog entry probably can&apos;t happen.  So that means the only case that&apos;s an issue is the case Nick describes, after an unsharded collection becomes sharded.  In that case we will still need to kick off a refresh of the CollectionMetadata somehow.&lt;/p&gt;</comment>
                            <comment id="1707494" author="spencer" created="Tue, 24 Oct 2017 16:27:35 +0000"  >&lt;p&gt;I believe the intent of that code was to handle the case where the shard has outdated CollectionMetadata - corresponding to a previous version of the collection - when the changeStream is established.  Since changeStreams are unversioned, if the first operation you perform on the newly sharded collection is opening a change stream the metadata won&apos;t be refreshed and you&apos;ll get a UUID mismatch between the sharding CollectionMetadata and the on-disk catalog.  If that happens we&apos;d like to refresh the CollectionMetadata and if the UUIDs now match continue without erroring to the user.&lt;/p&gt;</comment>
                            <comment id="1707485" author="nicholas.zolnierz" created="Tue, 24 Oct 2017 16:21:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;, yeah I changed to error code InvalidUUID for now to unblock the testing, but StaleEpoch works as well.  To give some more context on what&apos;s causing this, these are the steps:&lt;/p&gt;

&lt;p&gt;1. Open change stream cursor on unsharded collection&lt;br/&gt;
2. Shard the collection&lt;br/&gt;
3. Insert 2 documents, one for each shard&lt;br/&gt;
4. Assert that getNext() on the change stream cursor is able to see the insert&lt;/p&gt;

&lt;p&gt;I&apos;m not sure if this is expected to work or not, seems like a perfectly valid use case to me. &lt;/p&gt;</comment>
                            <comment id="1707414" author="esha.maharishi@10gen.com" created="Tue, 24 Oct 2017 15:50:48 +0000"  >&lt;p&gt;Ah yes, we should never uassert with a StaleConfig error code. In general, we should instead throw a StaleConfigException, so that it can include the version wanted and version received.&lt;/p&gt;

&lt;p&gt;However, I think this particular spot should uassert a different error code, probably StaleEpoch, since it&apos;s a UUID mismatch, which is equivalent to what an epoch mismatch meant before we had UUIDs. However, note that throwing a StaleEpoch will not cause the shard to refresh. What is supposed to happen if the change streams cursor&apos;s UUID does not match what&apos;s on the CollectionMetadata?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="440572">SERVER-31392</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="440626">SERVER-31394</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="451863">SERVER-31746</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="455254">SERVER-31847</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="450304">SERVER-31707</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 24 Oct 2017 15:50:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 29 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_14262" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>End date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 27 Oct 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-89</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 29 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>basit@voltaireapp.com</customfieldvalue>
            <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>nicholas.zolnierz@mongodb.com</customfieldvalue>
            <customfieldvalue>sjohnson540</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htiknb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htaebj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1952">Query 2017-11-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_14261" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>Start date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 24 Oct 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hti6r3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>