<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:04:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-43906] WiredTigerLAS.wt keeps growing daily</title>
                <link>https://jira.mongodb.org/browse/SERVER-43906</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I am running MongoDB 4.2.0 on&#160;Ubuntu 18.04.3 LTS, single server, no replicas. This is hosted in AWS on a&#160;m4.4xlarge instance type (16 cores, 64GB RAM) with 2.5TB volume (type io1, 8000 IOPS). The CloudWatch metrics look completely normal. CPU usage seems to be between 20-25%.&lt;/p&gt;

&lt;p&gt;After upgrading to 4.2.0 I started observing abnormal growth of&#160;WiredTigerLAS.wt. The only way to reclaim disk space is to restart the service. This needs to happen every day or two. In just under two days it has grown to 270GB. This can hardly be considered normal.&lt;/p&gt;

&lt;p&gt;I am willing to share diagnostics data to get to the bottom of this.&lt;/p&gt;

&lt;p&gt;I have found a &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40488&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;similar issue&lt;/a&gt; but it does not seem to apply. Also, I do not see a resolution in the existing issue.&lt;/p&gt;</description>
                <environment></environment>
        <key id="965528">SERVER-43906</key>
            <summary>WiredTigerLAS.wt keeps growing daily</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="aleksandarpuskas@gmail.com">Aleksandar Puskas</reporter>
                        <labels>
                    </labels>
                <created>Wed, 9 Oct 2019 05:58:27 +0000</created>
                <updated>Wed, 18 Dec 2019 08:53:10 +0000</updated>
                            <resolved>Wed, 18 Dec 2019 08:52:06 +0000</resolved>
                                    <version>4.2.0</version>
                                                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="2641327" author="dmitry.agranat" created="Wed, 18 Dec 2019 08:51:08 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, thank you for reporting this issue.&lt;/p&gt;

&lt;p&gt;I&apos;ll go ahead and close this ticket. As a reminder, we recommend to address the &quot;hot pages&quot; issue in your application to avoid the symptom of the reported LAS file overhead.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2636112" author="aleksandarpuskas@gmail.com" created="Tue, 17 Dec 2019 12:44:45 +0000"  >&lt;p&gt;Thank you. I suppose we can close the issue now. I really hope we are done.&lt;/p&gt;</comment>
                            <comment id="2636074" author="dmitry.agranat" created="Tue, 17 Dec 2019 12:05:25 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;Normally, we will try to reuse the free_space in LAS file, even under heavier workload, as can be seen from your latest data, we did this successfully for the last 4 days. &lt;/p&gt;

&lt;p&gt;I believe the issue with the unbound LAS file growth was addressed in 4.2.2, do you have any further questions about this issue?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;
</comment>
                            <comment id="2636048" author="aleksandarpuskas@gmail.com" created="Tue, 17 Dec 2019 11:29:36 +0000"  >&lt;p&gt;Hello Dima,&lt;/p&gt;

&lt;p&gt;Thank you for the explanation. Since the file will never shrink (just like before), does it mean it can still grow very large after even heavier workload? For instance, it has now reached 27GB. Can it grow beyond this if the database happens to be under even heavier write workload in say a few days?&lt;/p&gt;</comment>
                            <comment id="2636022" author="dmitry.agranat" created="Tue, 17 Dec 2019 10:49:00 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, thank you for keeping us up to date on this issue.&lt;/p&gt;

&lt;p&gt;After reviewing the latest data from 4.2.2, I can confirm that &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5196&quot; title=&quot;Data mismatch failures with test/checkpoint after enabling LAS sweep&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5196&quot;&gt;&lt;del&gt;WT-5196&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5220&quot; title=&quot;Re-enable LAS dropped table change from WT-5150&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5220&quot;&gt;&lt;del&gt;WT-5220&lt;/del&gt;&lt;/a&gt; fixed the issue of the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; unbound growth. &lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/240618/240618_SERVER-43906.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We can still see the same write-heavy (updates) workload putting a lot of pressure on the WT cache (as previously explained, due to hot pages). This, as expected, results in cache overflow mechanism kicking in to store historical document versions for all updates more recent than the oldest &#8220;history window&#8221; pinned in the cache. As you can see, the &lt;tt&gt;cache overflow table entries&lt;/tt&gt; metric grows until the end of the update workload. However, unlike before running on MongoDB 4.2.2, now we are &lt;b&gt;removing&lt;/b&gt; entries from the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; file, see &lt;tt&gt;cache overflow table remove calls&lt;/tt&gt; metric. &lt;/p&gt;

&lt;p&gt;What&apos;s more importantly, the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; on disk size does not grow unbound any more (stays flat), see &lt;tt&gt;cache overflow table on-disk size&lt;/tt&gt; metric, meaning that after removing some entries, we are successfully reusing LAS file&apos;s free_space.&lt;/p&gt;

&lt;p&gt;This is the expected behavior: the file should not continue to grow, even under heavy write workload, the file should reach it&apos;s steady state. It won&apos;t shrink, but it shouldn&apos;t grow either.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;
</comment>
                            <comment id="2635945" author="aleksandarpuskas@gmail.com" created="Tue, 17 Dec 2019 09:40:01 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I have just uploaded latest diagnostics data after the recent upgrade. Please let me know what you see. Thanks.&lt;/p&gt;</comment>
                            <comment id="2598614" author="dmitry.agranat" created="Wed, 11 Dec 2019 13:17:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, yes, you can stop the create/drop script on MongoDB 4.2.2. Please upload the usual data after a couple of days for us to review.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2598146" author="aleksandarpuskas@gmail.com" created="Wed, 11 Dec 2019 06:02:10 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I have just upgraded. I will watch this over the next few days and let you know. I sincerely hope this is resolved now.&lt;/p&gt;

&lt;p&gt;Shall I stop the periodical collection create/drop now? Thank you.&lt;/p&gt;</comment>
                            <comment id="2596125" author="dmitry.agranat" created="Tue, 10 Dec 2019 11:53:55 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/4.2/#dec-9-2019&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB 4.2.2&lt;/a&gt; was just released, could you try it and let us know if it fixes the issue?&lt;/p&gt;</comment>
                            <comment id="2571486" author="dmitry.agranat" created="Thu, 28 Nov 2019 13:39:55 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, both issues appear to be fixed in MongoDB 4.2.2.&lt;/p&gt;</comment>
                            <comment id="2563697" author="aleksandarpuskas@gmail.com" created="Sun, 24 Nov 2019 16:12:26 +0000"  >&lt;p&gt;Yes, it seems the file size is back to normal now. Is there any ETA for the resolution of this issue? Thank you.&lt;/p&gt;</comment>
                            <comment id="2563695" author="dmitry.agranat" created="Sun, 24 Nov 2019 16:08:36 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, based on the recent data, the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; file was removed after the process restart at &lt;tt&gt;2019-11-21T07:40:05&lt;/tt&gt;. Could you please check again?&lt;/p&gt;


&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/238203/238203_LAS_removed.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="2563679" author="aleksandarpuskas@gmail.com" created="Sun, 24 Nov 2019 14:16:07 +0000"  >&lt;p&gt;I have just uploaded new diagnostic data.&lt;/p&gt;</comment>
                            <comment id="2563673" author="dmitry.agranat" created="Sun, 24 Nov 2019 13:51:10 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;, looking at the diagnostic data you&apos;ve uploaded I do not see any reboot for the last 7 days (data ends by Oct 21st, 07:12)&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/238202/238202_uptime.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;Maybe it was done after Oct 21st, 07:12 UTC? If yes, could you upload the latest &lt;tt&gt;diagnostic.data&lt;/tt&gt; which will cover this? &lt;/p&gt;</comment>
                            <comment id="2557960" author="aleksandarpuskas@gmail.com" created="Thu, 21 Nov 2019 07:46:15 +0000"  >&lt;p&gt;Just so you know I have done an urgent reboot to get rid of the huge file but the file remained. It didn&apos;t shrink after service was restarted as would have hoped. How can I recycle the file?&lt;/p&gt;

&lt;p&gt;I have also uploaded the latest diagnostics data.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;&lt;del&gt;rw&lt;/del&gt;------ 1 mongodb mongodb 371129401344 Nov 20 21:23 WiredTigerLAS.wt&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="2545859" author="dmitry.agranat" created="Mon, 18 Nov 2019 08:39:17 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt; for keeping us up to date with the latest information. We are aiming to address this issue by &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5196&quot; title=&quot;Data mismatch failures with test/checkpoint after enabling LAS sweep&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5196&quot;&gt;&lt;del&gt;WT-5196&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5220&quot; title=&quot;Re-enable LAS dropped table change from WT-5150&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5220&quot;&gt;&lt;del&gt;WT-5220&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;
</comment>
                            <comment id="2542477" author="aleksandarpuskas@gmail.com" created="Fri, 15 Nov 2019 07:26:03 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I have just uploaded latest diagnostics file. Please note that today I observed a significant drop in&#160;WiredTigerLAS.wt file size. After being a couple of hundred gigabytes it has now shrunk to just below 17 gigabytes. This seems to have happened somewhere in the last 24-48 hours. Please let me know what you make of it. The temp collection drop hack is still in place but it didn&apos;t seem to matter when I first deployed it.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;&lt;del&gt;rw&lt;/del&gt;------ 1 mongodb mongodb 16950489088 Nov 15 06:53 WiredTigerLAS.wt&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="2516233" author="sue.loverso" created="Mon, 4 Nov 2019 15:59:51 +0000"  >&lt;p&gt;I will try to think of other suggestions. The tickets that are open to address this issue are &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5196&quot; title=&quot;Data mismatch failures with test/checkpoint after enabling LAS sweep&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5196&quot;&gt;&lt;del&gt;WT-5196&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5220&quot; title=&quot;Re-enable LAS dropped table change from WT-5150&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5220&quot;&gt;&lt;del&gt;WT-5220&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="2515417" author="dmitry.agranat" created="Sun, 3 Nov 2019 11:46:43 +0000"  >&lt;p&gt;Thanks Alexandar, we are looking at the latest data and will update you soon on our findings.&lt;/p&gt;</comment>
                            <comment id="2515383" author="aleksandarpuskas@gmail.com" created="Sun, 3 Nov 2019 07:22:16 +0000"  >&lt;p&gt;After only half a day of running a script that creates/drops a temp collection on a &quot;local&quot; database, I am seeing increase of&#160;WiredTigerLAS.wt size. Please see the before/after sizes below. Also, the matching diagnostics data was just uploaded.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;&lt;del&gt;rw&lt;/del&gt;------ 1 mongodb mongodb 257098989568 Nov 2 14:58 WiredTigerLAS.wt&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;&lt;del&gt;rw&lt;/del&gt;------ 1 mongodb mongodb 285380800512 Nov 3 07:14 WiredTigerLAS.wt&lt;/tt&gt;&lt;/p&gt;</comment>
                            <comment id="2515214" author="aleksandarpuskas@gmail.com" created="Sat, 2 Nov 2019 13:21:12 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I have just uploaded the latest diagnostics data. The file&#160;WiredTigerLAS.wt has grown to ~253GB. I will now let the suggested workaround run for a few days and report back. Thanks!&lt;/p&gt;</comment>
                            <comment id="2499241" author="dmitry.agranat" created="Thu, 24 Oct 2019 15:00:43 +0000"  >&lt;p&gt;Hi Alexandar, &lt;/p&gt;

&lt;p&gt;Before the upgrade to 4.2.1, we have observed the same behavior with the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; growth. Though the growth was more sustained, the file did grew to 5.5 GB. The biggest difference I can see, as compared to the previous workload, is that during this time, the issue with the &quot;hot documents&quot; was less severe. Specifically, the &lt;tt&gt;maximum page size at eviction&lt;/tt&gt; metric was almost always around 8 MB (as opposed to 73 MB as we&apos;ve seen before).&lt;/p&gt;

&lt;p&gt;After the upgrade, on 23rd, and up until today, the workload is almost idle so we do not see any issues.&lt;/p&gt;

&lt;p&gt;If the workload is back to execute heavy updates, I suggest implementing the above mentioned create/drop sequence and letting this run for a few days. We can review the data once again to determine the next steps.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima &lt;/p&gt;</comment>
                            <comment id="2499017" author="aleksandarpuskas@gmail.com" created="Thu, 24 Oct 2019 13:18:55 +0000"  >&lt;p&gt;I have just uploaded new data. Thanks Dima for all the hard work you are putting in to help me resolve this.&lt;/p&gt;</comment>
                            <comment id="2498669" author="dmitry.agranat" created="Thu, 24 Oct 2019 07:58:27 +0000"  >&lt;p&gt;Thank you for the update Alexandar. Could you upload the latest &lt;tt&gt;diagnostic.data&lt;/tt&gt; so that we can review and provide an up to date recommendations?&lt;/p&gt;</comment>
                            <comment id="2497522" author="aleksandarpuskas@gmail.com" created="Wed, 23 Oct 2019 14:56:15 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I can absolutely do this create/drop sequence but since the database has been really stable for at least a week or more, is it really necessary to start immediately or do we wait for the problems to begin first?&lt;/p&gt;</comment>
                            <comment id="2497315" author="dmitry.agranat" created="Wed, 23 Oct 2019 13:34:18 +0000"  >&lt;p&gt;Hi Aleksander,&lt;/p&gt;

&lt;p&gt;I completely understand the difficulties with the downgrade. I suggest we stay on 4.2 and continue the troubleshooting.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Are you saying that the workaround is to create and drop a temp collection every few minutes? Does the collection need to be in any specific database? Why does that help?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes, the workaround is to create and drop a temp collection every few minutes. This collection does not need to be in any specific database. This workaround would help us to understand if we are in the right direction with identifying the root cause and fixing this.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima &lt;/p&gt;</comment>
                            <comment id="2497279" author="aleksandarpuskas@gmail.com" created="Wed, 23 Oct 2019 13:15:39 +0000"  >&lt;p&gt;Hello Dima,&lt;/p&gt;

&lt;p&gt;Although there are backups of data before the upgrade, it is virtually impossible for me to get those backups up and running because the server is too critical to be able to play with it in such a fashion.&lt;/p&gt;

&lt;p&gt;If I was to spin a new server just to bring up old database, I wouldn&apos;t be able to replicate the load or the pattern that the live server is experiencing.&lt;/p&gt;

&lt;p&gt;I have been monitoring this issue closely and even though I could still see the problem a few days after I initially reported, for many days now the problem is gone sort of. The file has grown to a manageable size but stayed there for a long time. Today I did an upgrade to 4.2.1 and will continue to monitor for issues.&lt;/p&gt;

&lt;p&gt;Are you saying that the workaround is to create and drop a temp collection every few minutes? Does the collection need to be in any specific database? Why does that help?&lt;/p&gt;

&lt;p&gt;Thank you!&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2497065" author="dmitry.agranat" created="Wed, 23 Oct 2019 09:41:30 +0000"  >&lt;p&gt;Hi Aleksander,&lt;/p&gt;

&lt;p&gt;While we are waiting for data from the previous MongoDB version to determine if this is a regression, I&apos;ve mentioned that you might consider temporarily downgrading. There might be another way to workaround this issue on 4.2. The workaround is to create and drop some collection every 5 minutes.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;
</comment>
                            <comment id="2491225" author="keith.bostic" created="Sun, 20 Oct 2019 10:01:09 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We want them to create and drop some dummy table in a loop, let&apos;s say once every hour?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Yes, but I&apos;d do it more often than that, maybe once every 5 minutes?&lt;/p&gt;</comment>
                            <comment id="2488825" author="keith.bostic" created="Fri, 18 Oct 2019 10:56:47 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat&quot;&gt;dmitry.agranat&lt;/a&gt;, I&apos;m suspicious as well that something else is happening here other than &lt;a href=&quot;https://jira.mongodb.org/browse/WT-5150&quot; title=&quot;LAS sweep is not removing the entries that are no longer required&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-5150&quot;&gt;&lt;del&gt;WT-5150&lt;/del&gt;&lt;/a&gt;, but a steady cadence of creating and removing a file every few minutes should be a reasonably easy way to test that theory?&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;under heavy write workload, the file will never shrink (on disk). &lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;This isn&apos;t necessarily the case, but at the least the file should not continue to grow, that is, even under heavy write workload the file should reach steady state. It won&apos;t shrink, but it shouldn&apos;t grow either.&lt;/p&gt;</comment>
                            <comment id="2484316" author="dmitry.agranat" created="Wed, 16 Oct 2019 09:33:36 +0000"  >&lt;p&gt;Hi Aleksander,&lt;/p&gt;

&lt;p&gt;I am sorry to hear you are having these issues. In order for us to be able to determine if this is a regression, please provide the same data for a MongoDB version you&apos;ve used before moving to 4.2 (if it&apos;s still available).&lt;/p&gt;

&lt;p&gt;Just to summarize what we see based on the provided information.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/233856/233856_hot_pages.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Between A and D, we see two clusters of write conflict events happening between A-B and C-D. Between B and C, write conflicts are insignificant.&lt;/li&gt;
	&lt;li&gt;Write conflicts during these two events range between a few hundreds to a few thousands per second.&lt;/li&gt;
	&lt;li&gt;During these two time periods, A-B and C-D, we can see that the &lt;tt&gt;cache maximum page size at eviction&lt;/tt&gt; metric grows up to 73 MB. As mentioned earlier, this is an indication of a &quot;hot pages&quot; anti pattern during write conflicts. A write conflict occurs when WiredTiger detects concurrent writes to the same document. A conflicting write is retried until it completes without conflict.&lt;/li&gt;
	&lt;li&gt;We try hard to keep max&#160;page&#160;size&#160;at&#160;eviction&#160;low (around 8 MB), because larger pages take longer to evict. A 73 MB&#160;page would also need to get split into much more on-disk pages than a 8 MB&#160;page. The work required to go over that 73 MB&#160;page&#160;and reconstruct smaller pages, compressing them to put on disk. This is expensive and is CPU bound.&lt;/li&gt;
	&lt;li&gt;These two clusters of write conflicts events put lots of pressure on the wiredTiger cache, marked by the &lt;tt&gt;snapshot-window-settings current cache pressure percentage&lt;/tt&gt; metric, which causes us to start using the cache overflow mechanism. See the correlation with the &lt;tt&gt;cache overflow table insert calls&lt;/tt&gt; indicating we are forced to flush some pages to disk to keep the cache from getting stuck. As a result, the WiredTigerLAS.wt file or the corresponding &lt;tt&gt;cache overflow table on-disk size&lt;/tt&gt; metric above grows.&lt;/li&gt;
	&lt;li&gt;During this time, we can also see that it takes more time for checkpoints to complete as we are consistently pinning a large number of transactions into memory.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Alternatively, while we are waiting for data from the previous MongoDB version to determine if this is a regression, you might consider temporarily downgrading. If the data from the previous MongoDB version is not available any more, downgrading to that previous MongoDB version and providing us with the requested information will also help us to progress this investigation.&lt;/p&gt;

&lt;p&gt;Once we are able to prove or refute if this is a regression, we would also be able to answer the rest of the questions in your last comment.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;
</comment>
                            <comment id="2475729" author="aleksandarpuskas@gmail.com" created="Thu, 10 Oct 2019 14:44:44 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Is this introduced in 4.2.0? I do not recall having this issue before.&lt;/p&gt;

&lt;p&gt;Is there a way to turn this off or change behavior? Whatever the apps are doing (it&apos;s not a single app) is difficult to change now and caused no issues in the past versions.&lt;/p&gt;

&lt;p&gt;Also, I do not understand how 64GB RAM isn&apos;t enough. Any recommendations on how to improve hardware to stop this from happening?&lt;/p&gt;

&lt;p&gt;Does the file ever automatically shrink? I have noticed it goes away only after the service is restarted. I can hardly restart the service daily just to get rid of this cache file. This feels like a huge &lt;ins&gt;&lt;b&gt;bug&lt;/b&gt;&lt;/ins&gt;. Growing the file by 150GB each day cannot be called a feature. The apps depend heavily on MongoDB but this starts me thinking I might want to move away and look for other solutions.&lt;/p&gt;

&lt;p&gt;Can you please tell me why concurrent updates to the same document would be bad? Is there a document that advises against this practice?&lt;/p&gt;

&lt;p&gt;Do you have any particular doc id that I can take a look at?&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="2475559" author="dmitry.agranat" created="Thu, 10 Oct 2019 13:53:02 +0000"  >&lt;p&gt;Thanks Aleksander,&lt;/p&gt;

&lt;p&gt;The growth of the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; file is related to the Cache overflow feature. Cache overflow is kind of like swap for the WiredTiger cache, it utilizes disk to store updates to documents by flushing cache from memory to disk. In cases of extreme cache pressure, we begin using the cache overflow table which aims to prevent us from getting stuck.&lt;/p&gt;

&lt;p&gt;We store historical document versions for all updates more recent than the oldest &#8220;history window&#8221; pinned in the cache. From a quick review of the data provided, it looks like there are a lot of concurrent writes to the same document which creates this cache pressure, which is turn results in the &lt;tt&gt;WiredTigerLAS.wt&lt;/tt&gt; file growth.&lt;/p&gt;

&lt;p&gt;Could you review your application logic to figure out the reason behind these concurrent updates to the same document?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2475435" author="aleksandarpuskas@gmail.com" created="Thu, 10 Oct 2019 12:46:18 +0000"  >&lt;p&gt;I have just uploaded the requested files. Please note that I remove the logs regularly as they grow very big too. If you need more logs for investigative purposes I can collect them over the next days. Thank you.&lt;/p&gt;</comment>
                            <comment id="2475183" author="dmitry.agranat" created="Thu, 10 Oct 2019 07:07:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=aleksandarpuskas%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;aleksandarpuskas@gmail.com&quot;&gt;aleksandarpuskas@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/0e5609ae-a686-4761-a5e5-e15dddb4a6d0.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="988280">WT-5220</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="974989">WT-5196</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="988280">WT-5220</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="234111" name="LAS_remove.png" size="247829" author="dmitry.agranat@mongodb.com" created="Thu, 17 Oct 2019 18:13:22 +0000"/>
                            <attachment id="238203" name="LAS_removed.png" size="74180" author="dmitry.agranat@mongodb.com" created="Sun, 24 Nov 2019 16:07:03 +0000"/>
                            <attachment id="240618" name="SERVER-43906.png" size="94381" author="dmitry.agranat@mongodb.com" created="Tue, 17 Dec 2019 10:30:17 +0000"/>
                            <attachment id="233856" name="hot_pages.png" size="332008" author="dmitry.agranat@mongodb.com" created="Wed, 16 Oct 2019 08:55:59 +0000"/>
                            <attachment id="237543" name="nov15.png" size="323667" author="dmitry.agranat@mongodb.com" created="Mon, 18 Nov 2019 08:33:01 +0000"/>
                            <attachment id="236100" name="table_create_drop.png" size="339059" author="dmitry.agranat@mongodb.com" created="Sun, 3 Nov 2019 11:50:00 +0000"/>
                            <attachment id="238202" name="uptime.png" size="75409" author="dmitry.agranat@mongodb.com" created="Sun, 24 Nov 2019 13:48:38 +0000"/>
                            <attachment id="235384" name="writeConflicts.png" size="244268" author="dmitry.agranat@mongodb.com" created="Sun, 27 Oct 2019 09:08:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>35.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 10 Oct 2019 07:07:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 8 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 8 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aleksandarpuskas@gmail.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>keith.bostic@mongodb.com</customfieldvalue>
            <customfieldvalue>sue.loverso@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvwstr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvlbmf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvwf33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>