<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:00:00 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42256] Mongo unable to evict cache</title>
                <link>https://jira.mongodb.org/browse/SERVER-42256</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Yesterday, we had multiple slowdown on our primary server:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/224448/224448_Screenshot+from+2019-07-17+14-42-58.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;From our metrics, we can see that the cache evictions threads were not able to evict cache fast enough until the cache used reached 95% and that applications threads started to help the dedicated threads (source:&#160;&lt;a href=&quot;http://source.wiredtiger.com/mongodb-3.4/tune_cache.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://source.wiredtiger.com/mongodb-3.4/tune_cache.html&lt;/a&gt;):&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/224447/224447_Screenshot+from+2019-07-17+14-12-00.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;We already encountered this issue with mongo 3.2 two years ago (on the secondary server though) and the only thing that fixed the issue was to upgrade to mongo 3.4:&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-27700&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We plan to upgrade to mongo 3.6 in the following months but can you check that a regression has not be introduced in our current mongo version ?&lt;/p&gt;

&lt;p&gt;Another related ticket:&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27753&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-27753&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I can provide you diagnostic.data and slow log files. Thx for you help.&lt;/p&gt;</description>
                <environment></environment>
        <key id="859780">SERVER-42256</key>
            <summary>Mongo unable to evict cache</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="edwin.zhou@mongodb.com">Edwin Zhou</assignee>
                                    <reporter username="spuyet@gmail.com">S&#233;bastien Puyet</reporter>
                        <labels>
                    </labels>
                <created>Wed, 17 Jul 2019 12:51:18 +0000</created>
                <updated>Fri, 27 Oct 2023 20:42:46 +0000</updated>
                            <resolved>Mon, 26 Jul 2021 15:52:14 +0000</resolved>
                                    <version>3.4.16</version>
                                                    <component>Performance</component>
                    <component>Stability</component>
                    <component>WiredTiger</component>
                                        <votes>3</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="3963084" author="JIRAUSER1257066" created="Mon, 26 Jul 2021 15:52:04 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bigbourin%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;bigbourin@gmail.com&quot;&gt;bigbourin@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We appreciate you for following up and are happy to hear that you no longer are seeing the regression you earlier described. &lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3951453" author="bigbourin@gmail.com" created="Tue, 20 Jul 2021 08:31:39 +0000"  >&lt;p&gt;Hi !&lt;/p&gt;

&lt;p&gt;As per the following of our tests in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42698&quot; title=&quot;WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42698&quot;&gt;&lt;del&gt;SERVER-42698&lt;/del&gt;&lt;/a&gt; we finally managed to upgrade to 4.0.18 version which at least does not have the cache eviction regression introduced in 3.6.9, this is what I said in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42698&quot; title=&quot;WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42698&quot;&gt;&lt;del&gt;SERVER-42698&lt;/del&gt;&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;We recently were able to give 4.0.18 a try (to try to mitigate another issue which turned out to be even worse: &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5479&quot; title=&quot;Checkpoint schema lock can cause long stalls in command execution on systems with many active data handles&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5479&quot;&gt;&lt;del&gt;WT-5479&lt;/del&gt;&lt;/a&gt;) and we did not see any eviction performance issue on this version, so it looks like the regression introduce in 3.6.9 reported here (which was still present in 3.6.15) is NOT present in 4.0.18.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It&apos;s not clear if 4.0.18+ is better than before 3.6.9 in terms of cache eviction performance, but at least the biggest regression is not present in latest versions and we did not see any issue with cache eviction in our production environment. So I believe this ticket can be closed now ^^&lt;/p&gt;</comment>
                            <comment id="3950732" author="JIRAUSER1257066" created="Mon, 19 Jul 2021 21:37:32 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=adrien.jarthon%40ringcentral.com&quot; class=&quot;user-hover&quot; rel=&quot;adrien.jarthon@ringcentral.com&quot;&gt;adrien.jarthon@ringcentral.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your participation in helping us investigate and identify the regression you&apos;re seeing in 3.4. As you may already know, both versions 3.4 and 3.6 are EOL per our support policy. &lt;/p&gt;

&lt;p&gt;We&apos;re interested if you&apos;re seeing continued regressions in the eviction performance given later versions of MongoDB (v4.0 and later). If you&apos;re interested in continuing this investigation, can you please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and attach it to this ticket?&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="2505769" author="adrien.jarthon@ringcentral.com" created="Mon, 28 Oct 2019 22:37:18 +0000"  >&lt;p&gt;Well &quot;there have been many improvements made to the eviction server since&quot; is all relative as we are not even able to upgrade yet due to the drastic eviction performance decrease in 3.6.13. But yes I agree on the fact that the worse performance of 3.6 is more important to fix than the poor performance of 3.4 &#9786; We&apos;re currently spending some time testing various 3.6 versions to help you pinpoint the regression in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42698&quot; title=&quot;WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42698&quot;&gt;&lt;del&gt;SERVER-42698&lt;/del&gt;&lt;/a&gt; so we&apos;ll not spend too much time on 3.4 as it&apos;s mostly abandoned performance-wise anyway. Just hope we&apos;ll be able to upgrade at some point &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="2505704" author="daniel.hatcher" created="Mon, 28 Oct 2019 21:29:37 +0000"  >&lt;p&gt;As mentioned in an earlier comment by Dmitry, this specific case is the result of immense cache pressure possibly related to a bottleneck in the eviction server. Because this version of MongoDB is fairly old and there have been many improvements made to the eviction server since, I&apos;m not sure that there is a large amount of value in trying to resolve issues on this specific version. We have &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42698&quot; title=&quot;WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42698&quot;&gt;&lt;del&gt;SERVER-42698&lt;/del&gt;&lt;/a&gt; open to investigate possible issues with the eviction server on 3.6 so I propose we continue on that ticket and close this one. &lt;/p&gt;

&lt;p&gt;If you would like to continue looking at 3.4, I think the best path forward would be to compare two discrete 3.4.x versions where performance is &quot;acceptable&quot; on one patch version and &quot;not acceptable&quot; on another. However, our focus would still be on 3.6 and newer work.&lt;/p&gt;</comment>
                            <comment id="2362685" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 12:35:55 +0000"  >&lt;p&gt;And for the record I also created &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42662&quot; title=&quot;Memory leak in 3.6.13?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42662&quot;&gt;&lt;del&gt;SERVER-42662&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2362681" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 12:35:14 +0000"  >&lt;p&gt;I did this in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42698&quot; title=&quot;WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42698&quot;&gt;&lt;del&gt;SERVER-42698&lt;/del&gt;&lt;/a&gt; to separate the two issues as this one is related to 3.4.16 not so great eviction performance, whereas the new one is related to 3.6.13 regression on this matter.&lt;/p&gt;</comment>
                            <comment id="2362560" author="dmitry.agranat" created="Thu, 8 Aug 2019 11:27:38 +0000"  >&lt;p&gt;Thanks for the update Adrien, please upload the diagnostic.data from the primary and the secondary which was upgraded to 3.6.13 to the secure portal. Also, please provide the timestamps + timezone of the events you&apos;d like us to take a look.&lt;/p&gt;</comment>
                            <comment id="2362423" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 08:51:15 +0000"  >&lt;p&gt;We added some charts to our monitoring about this since we&apos;re hitting the limits of the eviction server (the single threaded one filling the queue apparently). On this server in 3.4.13 we regularly hit 30k+ page queued for eviction per second, and at this point it&apos;s pretty much saturated and can&apos;t fill up the queue any more, increasing cache usage to 95% and generating app thread evictions:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/226545/226545_screenshot-2.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="2354831" author="bigbourin@gmail.com" created="Thu, 1 Aug 2019 18:54:52 +0000"  >&lt;p&gt;Ok thanks, we started by updating the secondary to 3.6.13 and so far it&apos;s not looking good, even on the secondary which never had any eviction performance issue with 3.4, after one day with 3.6.13 we saw some spikes to 95% cache usage:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/225972/225972_screenshot-1.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This is the MAX cache usage for primary (yellow, 3.4) and secondary (green, 3.6.13 since the drop to 0 visible on the chart). We can clearly see on the secondary it used to be OK and start showing some dangerous spikes since the update, while the primary is actually close to dying every day because of this. We&apos;ll monitor for a couple more days but it doesn&apos;t look very safe to update in these conditions...&lt;/p&gt;</comment>
                            <comment id="2353516" author="dmitry.agranat" created="Thu, 1 Aug 2019 06:15:00 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;The eviction server is single-threaded. It&apos;s performance, contention and issues need to be carefully diagnosed in order to get to the root cause. There is no single metric which can surface eviction server efficiency and point to the root cause. You can try monitoring the &lt;tt&gt;wt cache pages evicted by application threads&lt;/tt&gt; metric which might be a good indication that the eviction is struggling and needs to be investigating.&lt;/p&gt;

&lt;p&gt;Just to reiterate, after the upgrade to the latest MongoDB version, please upload the diagnostic.data and we will be happy to take a look.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2352026" author="bigbourin@gmail.com" created="Wed, 31 Jul 2019 11:29:58 +0000"  >&lt;p&gt;Hi, so if I understand well (I&apos;m working with Sebastien) mongo has an &quot;eviction server&quot; which generates candidates for eviction or something like that? and then the multiple eviction threads (or application threads) evicts items from a queue which is filled by this servers? and so if the server is not fast enough the evictions waits (blocking the entire server if we&apos;re at 95%) for the eviction server?&lt;/p&gt;

&lt;p&gt;Is this a multi-threaded thing? As this is another mongoDB bottleneck we would need to monitor it closely, how can see how close we are to the eviction server saturation? If we can measure this we can also see how much of an improvement 3.6 brings in this regard.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="2351796" author="dmitry.agranat" created="Wed, 31 Jul 2019 07:59:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spuyet%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;spuyet@gmail.com&quot;&gt;spuyet@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Apologies for a delayed response. &lt;/p&gt;

&lt;p&gt;We can see that starting from 05:30 AM UTC, July 16th, there is a shift in the workload and the operations rate was gradually increased by ~x3.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/225709/225709_workload_shift.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;It looks to me like the main problem is that the eviction server is not queuing pages fast enough:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/225708/225708_eviction_server.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;This is a zoom in into the last our (12:05 - 13:05) of the above workload increase marked by &lt;b&gt;A-B&lt;/b&gt;&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;eviction calls to get a page found queue empty&lt;/tt&gt; is not as large as &lt;tt&gt;eviction calls to get a page&lt;/tt&gt;, so eviction workers are not starved for work all the time but still having trouble in ~27% of the time. In other words, if we had managed to evict pages at just a 0.27% higher rate, we would have kept cache utilization from rising to 95%.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;application&#160;thread&#160;time&#160;evicting&lt;/tt&gt; is small, &lt;tt&gt;application&#160;thread&#160;time&#160;waiting&#160;for&#160;cache&lt;/tt&gt; is large, indicating that&#160;application&#160;threads aren&apos;t finding eviction work queued for them so are&#160;waiting&#160;instead.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;eviction empty score&lt;/tt&gt; is large, meaning the eviction server often found the eviction queue empty, i.e. eviction workers are draining it faster than the server can fill it.&lt;/li&gt;
&lt;/ul&gt;


&lt;blockquote&gt;
&lt;p&gt;We plan to upgrade to mongo 3.6 in the following months but can you check that a regression has not be introduced in our current mongo version ?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I do not believe this is a regression of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700&quot; title=&quot;WT secondary performance drops to near-zero with cache full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27700&quot;&gt;&lt;del&gt;SERVER-27700&lt;/del&gt;&lt;/a&gt; which you have mentioned. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27700&quot; title=&quot;WT secondary performance drops to near-zero with cache full&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27700&quot;&gt;&lt;del&gt;SERVER-27700&lt;/del&gt;&lt;/a&gt; explicitly talks about a secondary member while here, we are investigating the Primary member during the reported event. In addition, we do not have the information of the issue you have encouraged 2 years ago to be able to determine if this is a regression.&lt;/p&gt;

&lt;p&gt;It will be difficult to determine which out of ~400 WT improvements we&apos;ve made since your current MongoDB version would help this workload with faster page eviction. We would be happy to re-evaluate this after the upgrade if the issue reoccurs. In addition, if possible, please consider upgrading to the latest of 4.0 (as of today, 4.0.11).&lt;/p&gt;


</comment>
                            <comment id="2347221" author="spuyet@gmail.com" created="Mon, 29 Jul 2019 07:49:07 +0000"  >&lt;p&gt;Dmitry any news about this issue ?&lt;/p&gt;</comment>
                            <comment id="2332328" author="spuyet@gmail.com" created="Thu, 18 Jul 2019 15:34:39 +0000"  >&lt;p&gt;The cache used started to grow more than 80% from the morning yes:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/224606/224606_Screenshot+from+2019-07-18+17-27-37.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;But we started to have a real impact on production between 2:00PM UTC+2 and 3:05PM UTC+2 (when we stepped down the server). Not sure that timezone really matters here though.&lt;/p&gt;</comment>
                            <comment id="2331937" author="dmitry.agranat" created="Thu, 18 Jul 2019 12:26:42 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spuyet%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;spuyet@gmail.com&quot;&gt;spuyet@gmail.com&lt;/a&gt;. Just to confirm, we are looking into the event that started with the workload increase/shift on July 16th at 06:00 AM and lasted until 01:00 PM UTC?&lt;/p&gt;


</comment>
                            <comment id="2331923" author="spuyet@gmail.com" created="Thu, 18 Jul 2019 12:14:20 +0000"  >&lt;p&gt;Hello Dmitry,&lt;/p&gt;

&lt;p&gt;I&apos;ve just uploaded the diagnostic data and the slow log files.&lt;/p&gt;

&lt;p&gt;We don&apos;t see anything different than other days.&lt;/p&gt;</comment>
                            <comment id="2331507" author="dmitry.agranat" created="Thu, 18 Jul 2019 07:15:21 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spuyet%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;spuyet@gmail.com&quot;&gt;spuyet@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report, we&apos;ll be happy to look into this.&lt;/p&gt;

&lt;p&gt;Has anything changed on the application side in terms of the workload prior to this event?&lt;/p&gt;

&lt;p&gt;Please upload the diagnostic data, mongod logs as well as the exact time and timezone of the reported issue.&lt;br/&gt;
You can upload all the data into &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/37be6a35-e2ec-432b-ad1f-f83ac4af1ca3.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this secure&lt;/a&gt; portal.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2330166" author="spuyet@gmail.com" created="Wed, 17 Jul 2019 12:56:48 +0000"  >&lt;p&gt;An interesting thing to notice is that after the steDdown the server was back to a &quot;normal&quot; state.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="887220">SERVER-42698</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="224447" name="Screenshot from 2019-07-17 14-12-00.png" size="35987" author="spuyet@gmail.com" created="Wed, 17 Jul 2019 12:45:52 +0000"/>
                            <attachment id="224448" name="Screenshot from 2019-07-17 14-42-58.png" size="17418" author="spuyet@gmail.com" created="Wed, 17 Jul 2019 12:43:09 +0000"/>
                            <attachment id="224606" name="Screenshot from 2019-07-18 17-27-37.png" size="41649" author="spuyet@gmail.com" created="Thu, 18 Jul 2019 15:33:27 +0000"/>
                            <attachment id="225708" name="eviction_server.png" size="381286" author="dmitry.agranat@mongodb.com" created="Wed, 31 Jul 2019 08:44:15 +0000"/>
                            <attachment id="225972" name="screenshot-1.png" size="105332" author="bigbourin@gmail.com" created="Thu, 1 Aug 2019 18:51:34 +0000"/>
                            <attachment id="226545" name="screenshot-2.png" size="107643" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 08:48:08 +0000"/>
                            <attachment id="225709" name="workload_shift.png" size="200279" author="dmitry.agranat@mongodb.com" created="Wed, 31 Jul 2019 08:44:11 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 18 Jul 2019 07:15:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 28 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 28 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bigbourin@gmail.com</customfieldvalue>
            <customfieldvalue>adrien.jarthon@ringcentral.com</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>spuyet@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvesz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hv3uzb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvef8f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>