<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:01:11 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42698] WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)</title>
                <link>https://jira.mongodb.org/browse/SERVER-42698</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While investigating &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42256&quot; title=&quot;Mongo unable to evict cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42256&quot;&gt;&lt;del&gt;SERVER-42256&lt;/del&gt;&lt;/a&gt; we tried 3.6.13 as suggested (only secondary for now) and so far we noticed some memory leak (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42662&quot; title=&quot;Memory leak in 3.6.13?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42662&quot;&gt;&lt;del&gt;SERVER-42662&lt;/del&gt;&lt;/a&gt;) and some &lt;b&gt;huge&lt;/b&gt; cache eviction server performance regression (we think).&lt;/p&gt;

&lt;p&gt;We added some charts to our monitoring about this since we&apos;re hitting the limits of the eviction server (the single threaded one filling the queue apparently). On one of our primary in 3.4.13 we regularly hit 30k+ page queued for eviction per second, and at this point it&apos;s pretty much saturated and can&apos;t fill up the queue any more, increasing cache usage to 95% and generating app thread evictions (this is &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42256&quot; title=&quot;Mongo unable to evict cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42256&quot;&gt;&lt;del&gt;SERVER-42256&lt;/del&gt;&lt;/a&gt;):&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/226547/226547_image-2019-08-08-10-46-36-023.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;So then we put a secondary on 3.6.13, nothing bad so far except that we already see some big cache used spikes with almost no traffic (only mongodump) as mentioned in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42256?focusedCommentId=2354831&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2354831&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-42256?focusedCommentId=2354831&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2354831&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;But then more recently we started creating new indexes on our 2 RS (one is 3.4.16 only, and the other has the secondary on 3.6.13) and that&apos;s when the disaster came in, basically the 3.6.13 secondary was queuing page for eviction more than 10 times slower than that (looks capped at 3k pages/sec) even though the pressure generated by the index creation is bigger, meaning 95+ cache use pretty quickly whereas the 3.4.16 secondary was performing correctly.&lt;/p&gt;

&lt;p&gt;We can see this well on this chart where we tried creating a couple indexes one by one on 1 RS then the other:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/226546/226546_image-2019-08-07-18-30-35-724.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Here we can see in green the 3.6.13 secondary and in yellow the 3.4.16 secondary.&lt;br/&gt;
At 17:43 and 17:45 for example we created 2 identical index on similar collections, generating approximately the same pressure on the cache (in MB/s read), and we can see that 3.4.13 managed to queue pages for eviction at the same speed and thus keep the cache at 80% without trouble, whereas the 3.6.13 server looks capped at ~3k page/sec and this evict much slower than the cache fills up.&lt;/p&gt;

&lt;p&gt;This clearly looks like a huge performance regression to me, and it also mean we can&apos;t even risk putting 3.6.13 in production as it&apos;ll probably die the second we do it don&apos;t you think?&lt;/p&gt;

&lt;p&gt;You have the diagnostic.data already uploaded for these 2 servers during this operation in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42662&quot; title=&quot;Memory leak in 3.6.13?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42662&quot;&gt;&lt;del&gt;SERVER-42662&lt;/del&gt;&lt;/a&gt;, it was yesterday (August 7th) at 17:40 UTC+2&lt;/p&gt;</description>
                <environment></environment>
        <key id="887220">SERVER-42698</key>
            <summary>WT Cache eviction server performance regression in 3.6.13 (compared to 3.4.16)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="rachelle.palmer@mongodb.com">Rachelle Palmer</assignee>
                                    <reporter username="bigbourin@gmail.com">Adrien Jarthon</reporter>
                        <labels>
                    </labels>
                <created>Thu, 8 Aug 2019 09:05:13 +0000</created>
                <updated>Fri, 27 Oct 2023 20:42:44 +0000</updated>
                            <resolved>Wed, 29 Jul 2020 14:38:04 +0000</resolved>
                                    <version>3.6.13</version>
                                                                        <votes>5</votes>
                                    <watches>26</watches>
                                                                                                                <comments>
                            <comment id="3202398" author="alexander.gorrod" created="Wed, 10 Jun 2020 05:53:52 +0000"  >&lt;p&gt;That is great news - thanks for circling back around &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=adrien.jarthon%40ringcentral.com&quot; class=&quot;user-hover&quot; rel=&quot;adrien.jarthon@ringcentral.com&quot;&gt;adrien.jarthon@ringcentral.com&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3196910" author="adrien.jarthon@ringcentral.com" created="Tue, 9 Jun 2020 12:44:16 +0000"  >&lt;p&gt;I have good news for you, we recently were able to give 4.0.18 a try (to try to mitigate another issue which turned out to be even worse: &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5479&quot; title=&quot;Checkpoint schema lock can cause long stalls in command execution on systems with many active data handles&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5479&quot;&gt;&lt;del&gt;WT-5479&lt;/del&gt;&lt;/a&gt;) and we did not see any eviction performance issue on this version, so it looks like the regression introduce in 3.6.9 reported here (which was still present in 3.6.15) is NOT present in 4.0.18.&lt;/p&gt;</comment>
                            <comment id="2583422" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:50:35 +0000"  >&lt;p&gt; Hi ! I&apos;m working with Adrien and we recently put mongo 3.6.15 to the same test we used for other versions:&lt;/p&gt;

&lt;p&gt;Just as a reminder here&apos;s exactly what we did:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Create an index on a medium collection (~6M records)&lt;/li&gt;
	&lt;li&gt;Create an index on a bigger collection (~21M records)&lt;/li&gt;
	&lt;li&gt;Launch mongodump for a couple of minutes (~15 minutes)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Please note that during this test the node that was using version 3.6.15 was secondary and was hidden.&lt;br/&gt;
 Here are the cache related graph during our test:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239284/239284_index_creation_mongo_3_6_15_secondary.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;br/&gt;
 &#160;&lt;br/&gt;
 We can see that during both index creations the cache used goes over 80% without the eviction going hard enough to keep it at 80% (like it did in 3.4.16).&lt;br/&gt;
 The interesting part happened during the mongodump (the last big cache used bump):&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;As soon as it started the cache used skyrocketed to 87.5% and didn&apos;t go back to 80% until I stop it&lt;/li&gt;
	&lt;li&gt;The cache activity was pretty high too but the eviction wasn&apos;t following along (not enough to keep the cache used at 80%)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Since it behaved much better than 3.6.13 with the same test we were confident enough to try it as primary in production so we did it on a Sunday afternoon when we had less traffic than usual and here&apos;s what happened:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239285/239285_mongo3_6_15_primary_cache.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239286/239286_mongo3_6_15_primary_ios.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;As soon as our 3.6.15 node was elected primary we can see that the cache used reached 87.5% pretty fast and stayed there for a really long time (~40 minutes), the cache activity was super high BUT the page queued for eviction was too low compared to the cache pressure we had at that moment. You can see how the 3.4.16 node responded to the (almost) same cache pressure when we put it back as primary.&lt;/p&gt;

&lt;p&gt;The IOs were also heavily used during that time (I assume this is a consequence of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4869&quot; title=&quot;Stop adding cache pressure when eviction is falling behind&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4869&quot;&gt;&lt;del&gt;WT-4869&lt;/del&gt;&lt;/a&gt; which, from what I understood, prevent read pages from being cached if cache used is at 87.5% or more)&lt;/p&gt;

&lt;p&gt;It shows that the eviction performance regression is still present in 3.6.15 but it&apos;s mitigated by the fact that from 87.5% cache used mongo is not caching read pages anymore and relies on disk IOs instead.&lt;/p&gt;

&lt;p&gt;We have recurrent jobs that are performed each hour at around :20 that create significant cache pressure (that 3.4.16 handles easily whereas 3.6.15 cannot keep its cache used to 80% during that time), here are some graph that shows some occurrence of these recurrent jobs and their effect on a 3.4.16 primary node (in yellow) and on our 3.6.15 when it was primary (in green):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239289/239289_mongo_3_4_16_vs_3_6_15_cache_cron.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;An interesting thing to look at as well is the IOs during these cache usage spikes (the flat purple line is 3.4.16, the blue one is 3.6.15):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239290/239290_mongo_3_4_16_vs_3_6_15_ios_cron.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;


&lt;p&gt; The good news is that we were able to use mongo 3.6.8 as primary in production as of yesterday and it behaves way better than more recent 3.6 versions (it also confirms what Adrien said in a previous comment: the cache eviction performance regression has been introduced in 3.6.9), here are some graphs showing that 3.6.8 behaves as good as 3.4.16 did (the primary was 3.4.16 until we switched to 3.6.8 the 12/3 at 11:00PM):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/239296/239296_mongo_3_6_8_cache2.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&lt;br class=&quot;atl-forced-newline&quot; /&gt;
We also noticed that the page requested from cache has almost doubled between 3.4.16 and 3.6.8, is that normal ? It didn&apos;t degraded the cache eviction performance but we don&apos;t understand why there&apos;s such a big difference.&lt;br/&gt;
I&apos;m currently uploading diagnostic data for both nodes (the 3.6.15 one and the 3.4.16) on the portal you provided in the first comment&lt;/p&gt;</comment>
                            <comment id="2506722" author="daniel.hatcher" created="Tue, 29 Oct 2019 15:31:07 +0000"  >&lt;p&gt;I had been doing some testing on my own of the difference between 3.6.8 and 3.6.9 and that picture was part of a question I was asking internally. Unfortunately, while comments can be made private pictures cannot which can cause a lack of context. In that screenshot we can see a distinct difference in how WiredTiger is treating eviction but there were a large package of WiredTiger-related changes between those two versions (&lt;a href=&quot;https://jira.mongodb.org/issues/?jql=project%20%3D%20WT%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%203.6.9&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;122 in 3.6.9&lt;/a&gt;). The specific ticket(s) that triggered my change in patterns may not be the same as yours so I did not want to present the screenshot as absolute proof of something wrong. I believe Brian has the right tact here; if we focus our investigation on the other large set of changes in 3.6.14 (and the upcoming 3.6.15), we can make progress that gives you the best performance moving forward rather than trying to recreate decent performance in the past.&lt;/p&gt;</comment>
                            <comment id="2505771" author="adrien.jarthon@ringcentral.com" created="Mon, 28 Oct 2019 22:39:36 +0000"  >&lt;p&gt;Thanks, I saw a new chart uploaded showing 3.6.8 and 3.6.9 without comment, is there anything you noticed here? are you not looking at the changes between both version which seem to have introduce most of the regression? We&apos;re still testing other versions and will be able to provide more feedback later, we&apos;ll add 3.6.15 to the test when it&apos;s available!&lt;/p&gt;</comment>
                            <comment id="2503122" author="brian.lane" created="Mon, 28 Oct 2019 04:11:20 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bigbourin%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;bigbourin@gmail.com&quot;&gt;bigbourin@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the nice writeup of this issue.  We have done some recent performance optimizations with eviction in newer releases, which is starting to show up in some of the 3.6.x releases now.  3.6.14 had some of these optimizations, and there are a couple more targetting the 3.6.15 release.  I would be interested in getting your feedback on any testing you could do once 3.6.15 is released.&lt;/p&gt;

&lt;p&gt;-Brian&lt;/p&gt;</comment>
                            <comment id="2419366" author="bigbourin@gmail.com" created="Fri, 13 Sep 2019 13:34:42 +0000"  >&lt;p&gt;Hi again, so today I gave a quick try at 3.6.8 with a small index creation and noticed it went pretty good with no significant increase in case use so similar behavior to 3.4.16 for me, which would mean the change is between 3.6.8 and 3.6.13. Then I tried 3.6.9 because there&apos;s a lot of changes in WT between the two and I already noticed quite a visible difference: &lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 3.6.8 &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; 3.6.9 &lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/230708/230708_screenshot-2.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/230707/230707_screenshot-3.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;In both these test I did the same thing:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The secondary as hidden but still had production replication traffic.&lt;/li&gt;
	&lt;li&gt;Then it&apos;s re-started in version 3.6.x&lt;/li&gt;
	&lt;li&gt;Then I run a mongodump on it to fill up the cache to 80%&lt;/li&gt;
	&lt;li&gt;An existing index is dropped and re-created on the primary, the spikes visible in these charts are the index creation on the secondary.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;3.6.9 is creating the index a bit faster than 3.6.8 but the eviction can&apos;t keep up and cache size increase. Not a lot in this example because it&apos;s a small index but you can imagine with a bigger one it&apos;s like my previous examples. I wanted to try with a bigger index after that to show you more data but unfortunately it killed my primary which couldn&apos;t keep up with the index creation and live traffic at the same time (not really what I call &quot;background&quot; index creation but that&apos;s another story &#9786;)&lt;/p&gt;

&lt;p&gt;I&apos;ll make more test, if you think of particular things I should try let me know, but it already looks like there&apos;s at least a good part of the regression between 3.6.8 and 3.6.9 to me.&lt;/p&gt;</comment>
                            <comment id="2418022" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 15:01:38 +0000"  >&lt;blockquote&gt;
&lt;p&gt;In the new data there is considerable cache pressure on the 3.6 node, seen in &quot;pages read into cache&quot;, that was not present before it was primary nor on the 3.4 node when it was primary, probably because a node that has been secondary for an extended period will have to warm the cache for the primary query load when it becomes primary. This difference in conditions between the 3.4 node and the 3.6 node makes it a little difficult to know whether the issue seen in the Aug 11th data on 3.6 was due to this higher cache pressure, or whether it was due to a difference between 3.4 and 3.6.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes of course it has much bigger cache activity after the switch because the cache wasn&apos;t warm, though these stay reasonable for this machine and we&apos;ve seen much bigger numbers on 3.4 with no eviction issue, for example this is what we have every week day on our 3.4 primary:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230619_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230619/230619_screenshot-6.png&quot; title=&quot;screenshot-6.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230619&quot; file-preview-title=&quot;screenshot-6.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230619/_thumb_230619.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
The cache read activity here is, for hours, above the levels 3.6 couldn&apos;t even cope with during 5 minutes.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;You mentioned above some tests with different workloads where no difference was seen between 3.6 and 3.4. Do you still have the diagnostic.data for those tests that you can upload, together with details (where and when, exactly)? We can check that to see if the hypothesis about a larger &quot;pages requested from cache&quot; being involved is supported by that data.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I don&apos;t have the diagnostic data for these tests any more, but I still have grafana instrumentation (not for long, 45 days retention) to which I could add the &quot;pages requested from cache&quot;. Here is it for 3.6:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230623_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230623/230623_screenshot-7.png&quot; title=&quot;screenshot-7.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230623&quot; file-preview-title=&quot;screenshot-7.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230623/_thumb_230623.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
And for 3.4:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230624_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230624/230624_screenshot-8.png&quot; title=&quot;screenshot-8.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230624&quot; file-preview-title=&quot;screenshot-8.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230624/_thumb_230624.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
Numbers looks similar, but as my test failed to reproduce the issue we have in production that&apos;s kind of expected.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;In addition, do you think it would be possible to add &quot;pages requested from cache&quot; to your monitoring and do some further experiments to see if a higher value of &quot;pages requested from cache&quot; is well correlated with the lower eviction efficiency that you are seeing? It would be ideal if it were possible to isolate a specific workload that triggered this behavior, such that we could reproduce it internally, although I understand that could be difficult.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;So with this new number I took more screenshot of previous events if it is of any help.&lt;br/&gt;
Here is the first event (August 7th) comparing index creation on 3.4 secondary and 3.6 secondary:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230627_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230627/230627_screenshot-9.png&quot; title=&quot;screenshot-9.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230627&quot; file-preview-title=&quot;screenshot-9.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230627/_thumb_230627.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;And this is the August 11th even when we tried 3.6 as primary:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230628_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230628/230628_screenshot-10.png&quot; title=&quot;screenshot-10.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230628&quot; file-preview-title=&quot;screenshot-10.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230628/_thumb_230628.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I think we&apos;ll try installing other intermediary version like 3.6.8 to see if we can reproduce, that would probably help reduce the diff to find the cause. I have one way to reproduce this easily on our production secondary (which is to create indexes), but unfortunately when I tried to reproduce this with my script it didn&apos;t show the same effect so I&apos;m not sure how else I can manage to reduce the search area? &lt;/p&gt;</comment>
                            <comment id="2414485" author="bruce.lucas@10gen.com" created="Tue, 10 Sep 2019 14:30:00 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;Thanks for uploading the new data and sorry for the delay.&lt;/p&gt;

&lt;p&gt;In the new data there is considerable cache pressure on the 3.6 node, seen in &quot;pages read into cache&quot;, that was not present before it was primary nor on the 3.4 node when it was primary, probably because a node that has been secondary for an extended period will have to warm the cache for the primary query load when it becomes primary. This difference in conditions between the 3.4 node and the 3.6 node makes it a little difficult to know whether the issue seen in the Aug 11th data on 3.6 was due to this higher cache pressure, or whether it was due to a difference between 3.4 and 3.6.&lt;/p&gt;

&lt;p&gt;However there is one possibly interesting difference between 3.4 and 3.6 that we saw both in the Aug 11th and in the Aug 7th data: the 3.6 node, in spite of servicing a generally lower workload (presumably because of the high cache fill ratio), showed a much higher rate (&amp;gt; 2x) of &quot;pages requested from cache&quot;. We don&apos;t know the reason for this, but it seems possible that this could cause a difference in cache eviction efficiency.&lt;/p&gt;

&lt;p&gt;You mentioned above some tests with different workloads where no difference was seen between 3.6 and 3.4. Do you still have the diagnostic.data for those tests that you can upload, together with details (where and when, exactly)? We can check that to see if the hypothesis about a larger &quot;pages requested from cache&quot; being involved is supported by that data.&lt;/p&gt;

&lt;p&gt;In addition, do you think it would be possible to add &quot;pages requested from cache&quot; to your monitoring and do some further experiments to see if a higher value of &quot;pages requested from cache&quot; is well correlated with the lower eviction efficiency that you are seeing? It would be ideal if it were possible to isolate a specific workload that triggered this behavior, such that we could reproduce it internally, although I understand that could be difficult.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="2408747" author="bigbourin@gmail.com" created="Thu, 5 Sep 2019 13:24:50 +0000"  >&lt;p&gt;Sorry to bump this but did you start looking into it?&lt;/p&gt;</comment>
                            <comment id="2366374" author="bigbourin@gmail.com" created="Mon, 12 Aug 2019 11:18:30 +0000"  >&lt;p&gt;If it is of any help, we also notice this eviction performance issue at the end of the mongodump, for example this morning at 8am (UTC+2) we can clearly see it on mongo1.s2:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/226838/226838_screenshot-5.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Whereas on 3.4.16 there is no visible issue on mongo2.s1 and even on mongo1.s2 when it was running 3.4.16.&lt;br/&gt;
You have the log and diagnostic data covering many samples of this event (it&apos;s a daily dump) &lt;/p&gt;</comment>
                            <comment id="2366291" author="bigbourin@gmail.com" created="Mon, 12 Aug 2019 09:11:45 +0000"  >&lt;p&gt;Hi again, I&apos;m bringing more bad news, we tried yesterday switching the 3.6.13 node to primary to see if this issue maybe was secondary specific and that it would work fine as Primary but it didn&apos;t, as we expected it quickly rose up to 95% cache usage and stayed there without being able to evict fast enough while performance was degrading (app thread doing eviction). After 10 minutes we rolled back to 3.4.13.&lt;/p&gt;

&lt;p&gt;Here&apos;s how it looked in our grafana:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;226829_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/226829/226829_screenshot-1.png&quot; title=&quot;screenshot-1.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;226829&quot; file-preview-title=&quot;screenshot-1.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/226829/_thumb_226829.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;So the 3.6.13 node (mongo1.s2) was primary between 10:46 and 10:57 (UTC+2).&lt;br/&gt;
I uploaded the new diagnostic data archive (for all 4 servers but you&apos;re only interested in mongo1.s2 and mongo2.s2 here, *.s1 is a different RS) and server logs  for the two servers during this day.&lt;br/&gt;
I think this kind of rules out the &quot;secondary only&quot; regression &#9785;&lt;/p&gt;</comment>
                            <comment id="2364788" author="bigbourin@gmail.com" created="Fri, 9 Aug 2019 15:32:13 +0000"  >&lt;p&gt;For the record I&apos;ve done more tests to try to narrow down the possible explanations for this performance issue.&lt;br/&gt;
First I tried on our staging server to delete and re-create some indexes (while cache is already full to force some evictions) in multiple collections in parallel to create more pressure, but in these tests both versions (3.4.16 and 3.6.13) showed similar performances (good but means no help to pinpoint the issue).&lt;/p&gt;

&lt;p&gt;Here are some charts showing the cache numbers during my benchmark for 3.4.16:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;226743_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/226743/226743_image-2019-08-09-13-24-05-160.png&quot; title=&quot;image-2019-08-09-13-24-05-160.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;226743&quot; file-preview-title=&quot;image-2019-08-09-13-24-05-160.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/226743/_thumb_226743.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;br/&gt;
3.6.13:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230708_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230708/230708_screenshot-2.png&quot; title=&quot;screenshot-2.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230708&quot; file-preview-title=&quot;screenshot-2.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230708/_thumb_230708.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This test was with a very small max cache size (1G), I also tried with 50G but the results where similar.&lt;br/&gt;
Here is the script I used to start the 3 index creation in parallel (10s between each start):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#!/usr/bin/env ruby&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;require &apos;mongo&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;client = Mongo::Client.new([ &apos;localhost:27017&apos; ])&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db = client.use(&apos;database_name&apos;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# 3 big collections (13G) identical in content&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  [:perftest1, {texts: 1}],&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  [:perftest2, {texts: 1}],&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  [:perftest3, {texts: 1}],&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;].each_with_index do |(collection, index), n|&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  Thread.new {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    sleep 7 + n * 10&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    puts &quot;#{Time.now}: re-creating index #{index} on #{collection} in background&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    db[collection].indexes.drop_one( index ) rescue nil&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    db[collection].indexes.create_one( index, background: true )&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    puts &quot;#{Time.now}: index #{index} on #{collection} finished&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;end&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Then I also tried reproducing the issue on our production 3.6 secondary but without the queries (to eliminate that from the equation), so I marked the secondary as hidden and re-created one index on a 5M records collection to compare if the impact and eviction speed would be the same, and yes it&apos;s the same.&lt;/p&gt;

&lt;p&gt;With secondary queries load (yesterday):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;230707_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/230707/230707_screenshot-3.png&quot; title=&quot;screenshot-3.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;230707&quot; file-preview-title=&quot;screenshot-3.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/230707/_thumb_230707.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;Without secondary query load (today):&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;226746_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/226746/226746_screenshot-4.png&quot; title=&quot;screenshot-4.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;226746&quot; file-preview-title=&quot;screenshot-4.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/226746/_thumb_226746.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The longer tail visible on the first chart is actually a couple more indexes that were created just after, but basically the shape is exactly the same for the first big index create we can see here (basically the first spike in cache activity).&lt;br/&gt;
Of course before that spike the index is created on the primary (yellow on the left chart) with absolutely no visible impact on the cache use (3.4.16).&lt;/p&gt;

&lt;p&gt;So these 2 tests confirms IMO that:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The issue is not linked to our workload on the secondary&lt;/li&gt;
	&lt;li&gt;It&apos;s not a global &quot;wired tiger eviction is slower in 3.6&quot; problem as it looked quite ok in  isolated testing.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;So maybe something related to the RS mechanism? the replication load? something secondary specific?&lt;/p&gt;</comment>
                            <comment id="2363112" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 16:18:14 +0000"  >&lt;p&gt;Sure! I&apos;ve just uploaded the log files for mongo1.s2 and mongo2.s1 on August 7th. Ok no problem for the new portal !&lt;/p&gt;</comment>
                            <comment id="2363001" author="daniel.hatcher" created="Thu, 8 Aug 2019 15:26:16 +0000"  >&lt;p&gt;Hi Adrien,&lt;/p&gt;

&lt;p&gt;There is a difference in eviction patterns between your 3.4 and 3.6 servers but we&apos;ll have to dig into it more to determine the cause. Could you please upload the logs from the two servers covering your entire index creation test time? I&apos;d like to compare specific timestamps. &lt;/p&gt;

&lt;p&gt;You can use &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/f3153cc8-f3f8-4d4c-9841-b1fa3300bd22.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this unique portal&lt;/a&gt; for this ticket so we can keep files distinct as the investigation paths between this and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42662&quot; title=&quot;Memory leak in 3.6.13?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42662&quot;&gt;&lt;del&gt;SERVER-42662&lt;/del&gt;&lt;/a&gt; diverge. I&apos;ve already copied the diagnostics so you don&apos;t have to worry about them.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="859780">SERVER-42256</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="241377" name="1s2_v3.6.15_B-C.png" size="554138" author="nuno.costa@mongodb.com" created="Tue, 24 Dec 2019 14:22:40 +0000"/>
                            <attachment id="241378" name="2s2_v3.4.16_A-B.png" size="614751" author="nuno.costa@mongodb.com" created="Tue, 24 Dec 2019 14:22:40 +0000"/>
                            <attachment id="241379" name="2s2_v3.4.16_C-D.png" size="623086" author="nuno.costa@mongodb.com" created="Tue, 24 Dec 2019 14:22:40 +0000"/>
                            <attachment id="226653" name="Screen Shot 2019-08-08 at 3.55.23 PM.png" size="640860" author="daniel.hatcher@mongodb.com" created="Thu, 8 Aug 2019 19:55:36 +0000"/>
                            <attachment id="235112" name="Screen Shot 2019-10-23 at 12.44.55 PM.png" size="573097" author="daniel.hatcher@mongodb.com" created="Wed, 23 Oct 2019 16:47:58 +0000"/>
                            <attachment id="239604" name="Screenshot 2019-12-06 at 15.15.40.png" size="814942" author="nuno.costa@mongodb.com" created="Fri, 6 Dec 2019 16:16:44 +0000"/>
                            <attachment id="241380" name="diag_A-B-C-D.png" size="705488" author="nuno.costa@mongodb.com" created="Tue, 24 Dec 2019 14:22:40 +0000"/>
                            <attachment id="226662" name="eviction.png" size="617143" author="bruce.lucas@mongodb.com" created="Thu, 8 Aug 2019 20:37:21 +0000"/>
                            <attachment id="226546" name="image-2019-08-07-18-30-35-724.png" size="111221" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 08:57:44 +0000"/>
                            <attachment id="226547" name="image-2019-08-08-10-46-36-023.png" size="107643" author="bigbourin@gmail.com" created="Thu, 8 Aug 2019 08:46:37 +0000"/>
                            <attachment id="226743" name="image-2019-08-09-13-24-05-160.png" size="82059" author="bigbourin@gmail.com" created="Fri, 9 Aug 2019 15:03:53 +0000"/>
                            <attachment id="239284" name="index_creation_mongo_3_6_15_secondary.png" size="142904" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:19:18 +0000"/>
                            <attachment id="239285" name="mongo3_6_15_primary_cache.png" size="132743" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:20:57 +0000"/>
                            <attachment id="239286" name="mongo3_6_15_primary_ios.png" size="48642" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:21:05 +0000"/>
                            <attachment id="239289" name="mongo_3_4_16_vs_3_6_15_cache_cron.png" size="115942" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:28:40 +0000"/>
                            <attachment id="239290" name="mongo_3_4_16_vs_3_6_15_ios_cron.png" size="48405" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:30:01 +0000"/>
                            <attachment id="239293" name="mongo_3_6_8_cache.png" size="141101" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:33:36 +0000"/>
                            <attachment id="239296" name="mongo_3_6_8_cache2.png" size="130585" author="driss.tahraouimaldague+mongodbatlas@gmail.com" created="Wed, 4 Dec 2019 16:44:52 +0000"/>
                            <attachment id="226829" name="screenshot-1.png" size="354154" author="bigbourin@gmail.com" created="Mon, 12 Aug 2019 09:05:46 +0000"/>
                            <attachment id="230628" name="screenshot-10.png" size="91105" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 14:54:50 +0000"/>
                            <attachment id="230708" name="screenshot-2.png" size="61076" author="bigbourin@gmail.com" created="Fri, 13 Sep 2019 13:22:45 +0000"/>
                            <attachment id="226744" name="screenshot-2.png" size="85910" author="bigbourin@gmail.com" created="Fri, 9 Aug 2019 15:03:57 +0000"/>
                            <attachment id="230707" name="screenshot-3.png" size="62381" author="bigbourin@gmail.com" created="Fri, 13 Sep 2019 13:22:43 +0000"/>
                            <attachment id="226745" name="screenshot-3.png" size="90871" author="bigbourin@gmail.com" created="Fri, 9 Aug 2019 15:26:20 +0000"/>
                            <attachment id="226746" name="screenshot-4.png" size="83405" author="bigbourin@gmail.com" created="Fri, 9 Aug 2019 15:27:00 +0000"/>
                            <attachment id="226838" name="screenshot-5.png" size="36215" author="bigbourin@gmail.com" created="Mon, 12 Aug 2019 11:16:38 +0000"/>
                            <attachment id="230619" name="screenshot-6.png" size="90897" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 14:25:02 +0000"/>
                            <attachment id="230623" name="screenshot-7.png" size="80475" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 14:44:04 +0000"/>
                            <attachment id="230624" name="screenshot-8.png" size="72328" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 14:45:53 +0000"/>
                            <attachment id="230627" name="screenshot-9.png" size="97644" author="bigbourin@gmail.com" created="Thu, 12 Sep 2019 14:52:26 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000iQ8ICQA0, 5002K00000kqfYfQAI]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 8 Aug 2019 15:26:16 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 35 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 35 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bigbourin@gmail.com</customfieldvalue>
            <customfieldvalue>adrien.jarthon@ringcentral.com</customfieldvalue>
            <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>brian.lane@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>driss.tahraouimaldague+mongodbatlas@gmail.com</customfieldvalue>
            <customfieldvalue>rachelle.palmer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvjhnz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr72bz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvj3xb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>