<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:05:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44112] The ChangeStream test fails on the latest sharded cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-44112</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;There is one change stream JSON test that fails on the latest sharded cluster. Reproducible at least on c# and Java drivers.&lt;/p&gt;

&lt;p&gt;Json test: &lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/change-streams-errors.json#L76&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/change-streams-errors.json#L76&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It looks like the reason of it that the latest server doesn&apos;t throw `280` error for the test &quot;Change Stream should error when _id is projected out&quot; anymore: &lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/change-streams-errors.json#L103&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/change-streams-errors.json#L103&lt;/a&gt;&lt;br/&gt;
So, our test runner doesn&apos;t catch the expected error and then fails.&lt;/p&gt;

&lt;p&gt;The evergreen task: &lt;a href=&quot;https://evergreen.mongodb.com/task_log_raw/dot_net_driver_unsecure_tests__version~latest_os~windows_64_topology~sharded_cluster_auth~noauth_ssl~nossl_test_f809794cf52538bd3f6b5c5ba5b9a74b17257fb7_19_10_17_23_46_15/2?type=T#L2824&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://evergreen.mongodb.com/task_log_raw/dot_net_driver_unsecure_tests__version~latest_os~windows_64_topology~sharded_cluster_auth~noauth_ssl~nossl_test_f809794cf52538bd3f6b5c5ba5b9a74b17257fb7_19_10_17_23_46_15/2?type=T#L2824&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="973217">SERVER-44112</key>
            <summary>The ChangeStream test fails on the latest sharded cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="justin.seyster@mongodb.com">Justin Seyster</assignee>
                                    <reporter username="dmitry.lukyanov@mongodb.com">Dmitry Lukyanov</reporter>
                        <labels>
                    </labels>
                <created>Fri, 18 Oct 2019 18:55:56 +0000</created>
                <updated>Fri, 27 Oct 2023 13:53:02 +0000</updated>
                            <resolved>Wed, 20 Nov 2019 00:35:03 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2550797" author="justin.seyster" created="Wed, 20 Nov 2019 00:35:03 +0000"  >&lt;p&gt;I&apos;m closing this out with &quot;Works as Designed,&quot; but &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt; should fix the failing driver tests nonetheless. If that turns out not to be the case, please feel free to re-open or file a new ticket.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jmikola&quot; class=&quot;user-hover&quot; rel=&quot;jmikola&quot;&gt;jmikola&lt;/a&gt;, I filed SPEC-1506 with my suggestion for how to update the tests in a way that would also stop this failure. The best case solution would be to implement both SPEC-1506 and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2543398" author="jmikola@gmail.com" created="Fri, 15 Nov 2019 16:33:50 +0000"  >&lt;blockquote&gt;&lt;p&gt;My opinion is that we should update the test specification to add that, if the iterator function can return an empty batch, the test should continue iterating until it gets either a non-empty batch or an error. I&apos;m thinking of the line Dmitry quoted in the above comment. Jeremy Mikola, it looks like you added that language in SPEC-1308. I&apos;d be interested in your opinion of the proposed change.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;IIRC, SPEC-1308 was added specifically for the benefit of async drivers. The reporter originally opened this in regard to the Node driver, wherein I believe the iteration method may block until an error is returned or a document becomes available. If neither occurs, it may be possible for the tests to block indefinitely (or for a very long time) and negatively impact the test suite.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;More broadly, though, tests should essentially never be written such that they expect an event or a failure to occur in a specific change stream batch, or on a particular getMore&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This was brought up in several conversations I&apos;ve had with server engineers in the past year, and I understand the reasoning behind it. I don&apos;t see a universal solution that addresses the limitations of drivers with blocking iteration methods. Short of those drivers implementing some workaround to give up on blocking iteration (perhaps with some time out behavior), I think those driver authors should continue to amend the driver test plan with any advice as needed (akin to what was done in SPEC-1308).&lt;/p&gt;</comment>
                            <comment id="2541369" author="jeff.yemin" created="Thu, 14 Nov 2019 19:49:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt; your question is best addressed by the driver team, not the server team.&lt;/p&gt;</comment>
                            <comment id="2541289" author="dmitry.lukyanov" created="Thu, 14 Nov 2019 19:03:29 +0000"  >&lt;p&gt;Can we open a spec ticket to change the tests description?&lt;/p&gt;</comment>
                            <comment id="2539826" author="bernard.gorman" created="Wed, 13 Nov 2019 22:09:53 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=justin.seyster&quot; class=&quot;user-hover&quot; rel=&quot;justin.seyster&quot;&gt;justin.seyster&lt;/a&gt;, yep: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44356&quot; title=&quot;Change Stream latency is determined by config server write rate&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44356&quot;&gt;&lt;del&gt;SERVER-44356&lt;/del&gt;&lt;/a&gt;. Briefly, we now open an additional cursor on the config servers to monitor for new shards, and so the latency of change streams may be limited by the write rate on the config servers. We&apos;re looking into ways to minimise or alleviate this latency.&lt;/p&gt;

&lt;p&gt;More broadly, though, tests should essentially &lt;b&gt;never&lt;/b&gt; be written such that they expect an event or a failure to occur in a specific change stream batch, or on a particular &lt;tt&gt;getMore&lt;/tt&gt;. &lt;tt&gt;$changeStream&lt;/tt&gt; does not and cannot make any such guarantees; it only promises that events will arrive as quickly as possible and in the correct order. We&apos;ve run into this problem numerous times in our own tests and the solution has invariably been, as Dave posted above, to assert that the event arrives at some point in the near future rather than on batch N:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;assert.soon(() =&amp;gt; changeStream.hasNext());&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2539795" author="justin.seyster" created="Wed, 13 Nov 2019 21:54:44 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt; Sorry for the radio silence on this! After I was investigating on Thursday, I got busy with the Barony, as well as my own Evergreen failure which went critical.&lt;/p&gt;

&lt;p&gt;I spoke with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charlie.swanson&quot; class=&quot;user-hover&quot; rel=&quot;charlie.swanson&quot;&gt;charlie.swanson&lt;/a&gt; about Change Stream behavior, and he confirmed that the behavior we&apos;re observing is the intended behavior: a getMore on a tailable, await-data cursor should have a default timeout of 1sec, after which it returns an empty batch. I spent some time going through the spec, but I couldn&apos;t find any language describing this behavior, but I also couldn&apos;t find any language contradicting it. This is how we want change streams to behave though, because it gives the driver and client an updated post-batch resume token once a second.&lt;/p&gt;

&lt;p&gt;My opinion is that we should update the test specification to add that, if the iterator function can return an empty batch, the test should continue iterating until it gets either a non-empty batch or an error. I&apos;m thinking of the line Dmitry quoted in the above comment. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jmikola&quot; class=&quot;user-hover&quot; rel=&quot;jmikola&quot;&gt;jmikola&lt;/a&gt;, it looks like you added that language in SPEC-1308. I&apos;d be interested in your opinion of the proposed change.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bernard.gorman&quot; class=&quot;user-hover&quot; rel=&quot;bernard.gorman&quot;&gt;bernard.gorman&lt;/a&gt; Do you have any idea why &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt; might have changed the timing for this test? Even if we modify the test so that it passes again, I think we should address the performance change.&lt;/p&gt;

&lt;p&gt;EDIT: I just saw that there&apos;s already a perf ticket attached to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt;: BF-15221. It&apos;s likely that this issue has the same cause.&lt;/p&gt;</comment>
                            <comment id="2523729" author="justin.seyster" created="Fri, 8 Nov 2019 00:10:21 +0000"  >&lt;p&gt;Quick note for myself: I checked the Evergreen logs to help narrow down what changed.&lt;br/&gt;
The first task that failed with this error was using revision dab2ae229660af56a786a084ebc36666c4dd6a91 of the server.&lt;br/&gt;
The task before that (which succeeded) was using revision 7264dd03a981ea9cc86dcdc073e7a6dedaa995af.&lt;/p&gt;

&lt;p&gt;Update:&lt;br/&gt;
I did some more experimenting, and this test failure appears to be related to commit 97cc7b5838db4ef13ede3149c44bceca8f5c2977 of the server (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;When I run the Java driver version of this test against the version of the server previous to 97cc7b, the first getMore immediately returns the expected error. When I run the same test against 97cc7b, it takes multiple getMore commands to get a response that is not an empty batch.&lt;/p&gt;

&lt;p&gt;I also wanted to rule out commit f80979 of the &lt;em&gt;C# driver&lt;/em&gt; as a possible cause; that&apos;s the first commit in Evergreen where this failure occurs, and it seems to involve tailable cursors. I ran f80979 on Evergreen against an older version of the server, and it succeeds, meaning the failure can&apos;t be caused by a change in f80979.&lt;/p&gt;

&lt;p&gt;tl;dr: The failing test used to pass because it got its response in the first getMore. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42723&quot; title=&quot;New shard with new database can be ignored by change streams&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42723&quot;&gt;&lt;del&gt;SERVER-42723&lt;/del&gt;&lt;/a&gt; changed mongos/mongod behavior so that it now takes multiple getMore commands to get the expected error, and the C# driver is not designed to retry the getMore, which is the caues of this failure.&lt;/p&gt;</comment>
                            <comment id="2515129" author="justin.seyster" created="Sat, 2 Nov 2019 01:46:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt; Thanks for bearing with me! Good question about the spec. I&apos;m going to have to get back to you on that next week. I&apos;ll need to ask the team to figure out which spec doc is relevant here.&lt;/p&gt;

&lt;p&gt;I was basing my initial answer off my read of the source code. It&apos;s clear that it was intentionally designed for the getMore to time out with no results but also no error when when the mongos doesn&apos;t see any events from the shards for a period of time. The duration that the mongos waits is controlled by the &quot;awaitDataTimeout&quot; parameter to the getMore command (which is 1,000ms by default). My expectation is that setting a very high &quot;awaitDataTimeout&quot; value would get the test passing again. That&apos;s probably not the route we want to take, though.&lt;/p&gt;

&lt;p&gt;My reading of the linked test specification is that &quot;iterate changeStream once&quot; refers to calling the driver&apos;s next() function exactly once without necessarily specifying how many times the next() function issues getNext commands to the server.  I&apos;m not familiar with these documents, though, so it&apos;s definitely possible that I&apos;m misunderstanding.&lt;/p&gt;

&lt;p&gt;It sounds like the next step is to get on the same page about what the spec says for tailable+awaitData cursors timing out. I&apos;ll see what I can find next week. Query should also make sure to investigate if there&apos;s a performance regression in change streams here.&lt;/p&gt;

&lt;p&gt;Also, I misspoke in my previous message. I&apos;m doing my testing on the &quot;master&quot; branch, which &lt;em&gt;is&lt;/em&gt; 4.3, not 4.4. Sorry about that! If I do more testing, I&apos;ll try to do it on the specific commits you&apos;re using. Thanks again!&lt;/p&gt;</comment>
                            <comment id="2515089" author="dmitry.lukyanov" created="Fri, 1 Nov 2019 23:16:09 +0000"  >&lt;p&gt;&amp;gt;What manual code changes did you add?&#160;&lt;/p&gt;

&lt;p&gt;the manual change to be able to call a server `getMore` command more than once.&lt;/p&gt;

&lt;p&gt;&amp;gt;are you referring to a driver &quot;getMore&quot; function or the server &quot;getMore&quot; command?&lt;/p&gt;

&lt;p&gt;to the server &quot;getMore&quot; command&lt;/p&gt;

&lt;p&gt;&amp;gt;The client needs to keep calling &quot;getMore&quot; until it sees results.&lt;/p&gt;

&lt;p&gt;I&apos;m not familiar with this statement, can you point me out on the spec with it?&lt;/p&gt;

&lt;p&gt;&amp;gt;In the latter case, a &quot;getMore&quot; command that returns 0 results is considered correct behavior from the server&lt;/p&gt;

&lt;p&gt;yes, the successful `getMore` response contains no documents.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;NOTE: this is the description of the test for changeStreams: &lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/README.rst#spec-test-runner&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/specifications/blob/master/source/change-streams/tests/README.rst#spec-test-runner&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In particular, this line tells us to iterate changeStream only once:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;If there was no error and result.error is set, iterate changeStream once and capture any error.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2515073" author="justin.seyster" created="Fri, 1 Nov 2019 22:50:14 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt; Thanks for the additional info. Sorry, I&apos;m not sure what you mean. What manual code changes did you add? When you say that you call &quot;getMore&quot; twice, are you referring to a driver &quot;getMore&quot; function or the server &quot;getMore&quot; command?&lt;/p&gt;

&lt;p&gt;In the former case, a driver &quot;getMore&quot; function that returns 0 results may be a driver bug, depending on the driver specification. (I&apos;m not familiar with the C# driver specification.) In the latter case, a &quot;getMore&quot; command that returns 0 results is considered correct behavior from the server. The client needs to keep calling &quot;getMore&quot; until it sees results. When I ran the test locally using the Java driver, I saw that it would keep sending &quot;getMore&quot; commands forever if the server kept sending back empty responses.&lt;/p&gt;

&lt;p&gt;That said, if mongos used to reliably return the change in the first &quot;getMore&quot; batch but is now taking several seconds to return the result, that may be a real performance issue.&lt;/p&gt;

&lt;p&gt;By the way, my local testing was against 4.4 top-of-tree. I could try 4.3 next week to see if maybe that will reproduce the issue. Thanks for helping me investigate this!&lt;/p&gt;</comment>
                            <comment id="2515052" author="dmitry.lukyanov" created="Fri, 1 Nov 2019 22:29:54 +0000"  >&lt;p&gt;&amp;gt;Are you able to reproduce it locally?&lt;/p&gt;

&lt;p&gt;yes, checked now on the server versions `4.3.0-1818-g6ca9be7` and `4.3.0-2014-g8010c51`.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;As I understand the problem is related to this line: &lt;a href=&quot;https://github.com/mongodb/mongo-csharp-driver/blob/master/tests/MongoDB.Driver.Tests/Specifications/change-streams/tests/change-streams-errors.json#L105&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-csharp-driver/blob/master/tests/MongoDB.Driver.Tests/Specifications/change-streams/tests/change-streams-errors.json#L105&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;When I call `getMore` command, I don&apos;t see any errors. But then when I call `getMore` a second time (after manual code changes), the error with code 280 and message:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Command getMore failed: Encountered an event whose _id field, which contains the resume token, was modified by the pipeline. Modifying the _id field of an event makes it impossible to resume the stream from that point. Only transformations that retain the unmodified _id field are allowed.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;has been thrown.&lt;/p&gt;

&lt;p&gt;NOTE: the error is not reproducible on the server version `4.3.0-762-gf13641a`.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;UPDATE&lt;/b&gt;: The error can be thrown not necessarily after the second attempt. In my tests, I saw the third or the forth attempts when the error has been thrown&lt;/p&gt;</comment>
                            <comment id="2515018" author="justin.seyster" created="Fri, 1 Nov 2019 21:59:18 +0000"  >&lt;p&gt;After I posted the above comment, I remembered that mongod/mongos only log a COMMAND if it exceeds the &quot;slowms&quot; threshold, which is 100ms by default. It is possible that there is a third getMore call in each test that isn&apos;t getting logged because it returns right away, perhaps with nreturned &amp;gt; 0.&lt;/p&gt;

&lt;p&gt;When I have some time I also plan to look at the logs for one of the successful runs to see how many getMore calls it has to make before it sees the error.&lt;/p&gt;</comment>
                            <comment id="2513504" author="justin.seyster" created="Thu, 31 Oct 2019 23:34:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.lukyanov&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.lukyanov&quot;&gt;dmitry.lukyanov&lt;/a&gt; You mentioned that this reproduces on C# and Java. Are you able to reproduce it locally? Unfortunately, I was not able to make this happen using the Java test. (I didn&apos;t try to execute the C# test suite, because I don&apos;t have a Windows test machine readily available.)&lt;/p&gt;

&lt;p&gt;My initial thought was that maybe mongos was returning an empty batch in the getMore response (which is legal), causing the test to give up and fail. In my testing, though, the test keeps sending getMore requests when it gets empty batches from mongos, which is the correct behavior.&lt;/p&gt;

&lt;p&gt;In the logs from Evergreen, though, the test sends exactly two getMore commands and then gives up. (I&apos;m looking at the mongos logs from &lt;a href=&quot;https://evergreen.mongodb.com/task/dot_net_driver_secure_tests__version~latest_os~windows_64_topology~sharded_cluster_auth~auth_ssl~ssl_test_49d23aeaafc443a8ce04720c9c3c5c81de4c4321_19_10_29_16_19_55&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this failure&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;(async = true)&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./mci/8f82fe232dda6f5a77d481ac8871feff/drivers-tools/.evergreen/orchestration/db/mongo-65jwqy/mongos.log:2019-10-29T19:14:58.371+0000 I  COMMAND  [conn2171] command change-stream-tests.test command: getMore { getMore: 6199935151688990082, collection: &quot;test&quot;, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;328c2bd5-eb89-41b4-9bb3-8ea505c3059b&quot;) }, $clusterTime: { clusterTime: Timestamp(1572376497, 24), signature: { hash: BinData(0, 79F3B098B291669A30AE2504FD09FB07BE31D898), keyId: 6753301787618312232 } } } originatingCommand: { aggregate: &quot;test&quot;, pipeline: [ { $changeStream: {} }, { $project: { _id: 0 } } ], cursor: {}, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;328c2bd5-eb89-41b4-9bb3-8ea505c3059b&quot;) } } nShards:2 cursorid:6199935151688990082 numYields:0 nreturned:0 reslen:306 protocol:op_msg 1000ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./mci/8f82fe232dda6f5a77d481ac8871feff/drivers-tools/.evergreen/orchestration/db/mongo-65jwqy/mongos.log:2019-10-29T19:14:59.632+0000 I  COMMAND  [conn2173] command change-stream-tests.test command: getMore { getMore: 2518998828756750331, collection: &quot;test&quot;, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;07b8c00b-6bbe-4ea9-aa0b-b018bb41b937&quot;) }, $clusterTime: { clusterTime: Timestamp(1572376498, 34), signature: { hash: BinData(0, B297D66CED85ED2C9889BE8CDF2B93B35F696D85), keyId: 6753301787618312232 } } } originatingCommand: { aggregate: &quot;test&quot;, pipeline: [ { $changeStream: {} }, { $project: { _id: 0 } } ], cursor: {}, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;07b8c00b-6bbe-4ea9-aa0b-b018bb41b937&quot;) } } nShards:2 cursorid:2518998828756750331 numYields:0 nreturned:0 reslen:306 protocol:op_msg 1001ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;(async = false)&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./mci/8f82fe232dda6f5a77d481ac8871feff/drivers-tools/.evergreen/orchestration/db/mongo-65jwqy/mongos.log:2019-10-29T19:25:43.681+0000 I  COMMAND  [conn7919] command change-stream-tests.test command: getMore { getMore: 9096175620065252820, collection: &quot;test&quot;, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;2fd89215-7d08-4067-90c1-27912443afbe&quot;) }, $clusterTime: { clusterTime: Timestamp(1572377142, 60), signature: { hash: BinData(0, 19DB81E36C8F74DECE6442DC1F232F516E1F06C5), keyId: 6753301787618312232 } } } originatingCommand: { aggregate: &quot;test&quot;, pipeline: [ { $changeStream: {} }, { $project: { _id: 0 } } ], cursor: {}, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;2fd89215-7d08-4067-90c1-27912443afbe&quot;) } } nShards:2 cursorid:9096175620065252820 numYields:0 nreturned:0 reslen:306 protocol:op_msg 1001ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./mci/8f82fe232dda6f5a77d481ac8871feff/drivers-tools/.evergreen/orchestration/db/mongo-65jwqy/mongos.log:2019-10-29T19:25:44.979+0000 I  COMMAND  [conn7921] command change-stream-tests.test command: getMore { getMore: 3119740060742483673, collection: &quot;test&quot;, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;08c1b04c-7ed5-4fb1-83e5-18a94f52c9b8&quot;) }, $clusterTime: { clusterTime: Timestamp(1572377143, 34), signature: { hash: BinData(0, 4D757549276B6B6BA660E1E7F8E74BB2F4307AA2), keyId: 6753301787618312232 } } } originatingCommand: { aggregate: &quot;test&quot;, pipeline: [ { $changeStream: {} }, { $project: { _id: 0 } } ], cursor: {}, $db: &quot;change-stream-tests&quot;, lsid: { id: UUID(&quot;08c1b04c-7ed5-4fb1-83e5-18a94f52c9b8&quot;) } } nShards:2 cursorid:3119740060742483673 numYields:0 nreturned:0 reslen:306 protocol:op_msg 1001ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The two mysteries are:&lt;br/&gt;
1) Why is mongos taking so long to respond with the change? The &quot;await data&quot; timeout is 1 second, meaning that the mongos has no events for two full seconds. Not impossible, but longer than I would expect.&lt;br/&gt;
2) Why is the driver giving up? Dmitry, if you can reproduce this locally, would it be possible for you to use the debugger or logging to determine why the driver stops retrying? That would be my next step if I could reproduce this and/or if I had a better understanding of the C# driver.&lt;/p&gt;</comment>
                            <comment id="2498418" author="david.storch" created="Wed, 23 Oct 2019 23:21:16 +0000"  >&lt;p&gt;I tried to reproduce this using the server&apos;s infrastructure for testing sharded clusters, but the change stream failed with the expected code (code:280, codeName:ChangeStreamFatalError).&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;(function() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;use strict&quot;;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;let st = new ShardingTest({shards: 2});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;const testDb = st.s.getDB(&quot;test&quot;);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;st.shardColl(testDb.c, {_id: 1}, {_id: 0}, {_id: 0});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;let changeStream = testDb.c.watch([{$project: {_id: 0}}]);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;assert.writeOK(testDb.c.insert({_id: 1}));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;assert.soon(() =&amp;gt; changeStream.hasNext());&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}());&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;What&apos;s the topology of the sharded cluster used by the drivers test? Is there any special configuration of the cluster we need in order to repro this?&lt;/p&gt;</comment>
                            <comment id="2498382" author="kaitlin.mahar" created="Wed, 23 Oct 2019 22:30:08 +0000"  >&lt;p&gt;I believe the test expects the failure on the first getMore. The linked JSON test has drivers do the following:&lt;/p&gt;

&lt;p&gt;1) create a change stream, which sends the initial aggregate command&lt;/p&gt;

&lt;p&gt;2) do an insertOne to generate a new event&lt;/p&gt;

&lt;p&gt;3) try to iterate the change stream to get the new event. this will trigger a getMore, since the initial batch returned by aggregate was empty. the getMore returns the error.&lt;/p&gt;</comment>
                            <comment id="2498324" author="ian.boros" created="Wed, 23 Oct 2019 21:41:31 +0000"  >&lt;p&gt;Does this test expect the aggregation to fail immediately (as part of the &apos;aggregate&apos; command being sent)? Does it attempt to run any getMores? I cannot tell from the logs provided.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="888141">SERVER-42723</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 18 Oct 2019 20:24:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 12 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 12 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bernard.gorman@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>dmitry.lukyanov@mongodb.com</customfieldvalue>
            <customfieldvalue>ian.boros@mongodb.com</customfieldvalue>
            <customfieldvalue>jeff.yemin@mongodb.com</customfieldvalue>
            <customfieldvalue>jmikola@mongodb.com</customfieldvalue>
            <customfieldvalue>justin.seyster@mongodb.com</customfieldvalue>
            <customfieldvalue>kaitlin.mahar@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvy3on:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvb61z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3284">Query 2019-11-04</customfieldvalue>
    <customfieldvalue id="3285">Query 2019-11-18</customfieldvalue>
    <customfieldvalue id="3286">Query 2019-12-02</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvxpxz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>