<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:29:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32154] Unbounded lookaside table growth during insert workload</title>
                <link>https://jira.mongodb.org/browse/SERVER-32154</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;1-node repl set, SSD, 50 GB oplog, 4 GB cache, 5 threads inserting 130 kB docs.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/172228/172228_growth.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;It appears that the lookaside table, both number of entries and size of WiredTigerLAS.wt (&quot;csv size&quot; metric), grows without bound during a heavy write workload. Even after the workload is stopped the table doesn&apos;t shrink.&lt;/p&gt;

&lt;p&gt;FTDC data attached.&lt;/p&gt;</description>
                <environment></environment>
        <key id="467495">SERVER-32154</key>
            <summary>Unbounded lookaside table growth during insert workload</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                    </labels>
                <created>Sun, 3 Dec 2017 14:50:41 +0000</created>
                <updated>Mon, 30 Oct 2023 23:10:12 +0000</updated>
                            <resolved>Wed, 8 Aug 2018 15:35:20 +0000</resolved>
                                    <version>3.6.0-rc8</version>
                                    <fixVersion>3.6.6</fixVersion>
                    <fixVersion>4.0.0</fixVersion>
                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="1970392" author="bruce.lucas@10gen.com" created="Wed, 8 Aug 2018 15:34:38 +0000"  >&lt;p&gt;Yes, this workload is much better behaved now on 3.6.6. Some of the work was done in earlier tickets, but that is the max fix version on the linked tickets, so I think we should mark this as fixed in 3.6.6.&lt;/p&gt;</comment>
                            <comment id="1969483" author="thomas.schubert" created="Tue, 7 Aug 2018 18:25:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, has this issue been resolved or does there need to be additional work that needs to be done?&lt;/p&gt;</comment>
                            <comment id="1815492" author="michael.cahill" created="Sun, 25 Feb 2018 23:37:01 +0000"  >&lt;p&gt;I&apos;ve opened &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3937&quot; title=&quot;Tune lookaside sweep to react to workload&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3937&quot;&gt;&lt;del&gt;WT-3937&lt;/del&gt;&lt;/a&gt; for tuning of the lookaside sweep and linked it here.&lt;/p&gt;</comment>
                            <comment id="1814726" author="ian@10gen.com" created="Fri, 23 Feb 2018 19:53:56 +0000"  >&lt;p&gt;OK, I&apos;ve added the link to &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766&quot; title=&quot;Lookaside sweep for obsolete updates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3766&quot;&gt;&lt;del&gt;WT-3766&lt;/del&gt;&lt;/a&gt;.  And it would be good if &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt; could comment on whether we should file another WT ticket based on the linked comment?&lt;/p&gt;</comment>
                            <comment id="1814723" author="bruce.lucas@10gen.com" created="Fri, 23 Feb 2018 19:51:55 +0000"  >&lt;p&gt;I think this ticket needs both &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3911&quot; title=&quot;Ignore lookaside data on shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3911&quot;&gt;&lt;del&gt;WT-3911&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766&quot; title=&quot;Lookaside sweep for obsolete updates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3766&quot;&gt;&lt;del&gt;WT-3766&lt;/del&gt;&lt;/a&gt;. The latter is marked as fixed in 3.6.3, but I think it &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766?focusedCommentId=1814718&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1814718&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;might need more work&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1814392" author="ian@10gen.com" created="Fri, 23 Feb 2018 16:49:21 +0000"  >&lt;p&gt;Marking as fixversion=Backlog because there are no plans to work on this ticket directly and it will instead be resolved with the work on &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3911&quot; title=&quot;Ignore lookaside data on shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3911&quot;&gt;&lt;del&gt;WT-3911&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1801941" author="bruce.lucas@10gen.com" created="Sat, 10 Feb 2018 12:45:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;, following up on our conversation yesterday, I&apos;m still seeing the same shutdown behavior in 3.6.2 regarding lookaside table on shutdown - shutdown takes a very long time and uses a very large amount of memory, many times the size of the cache.&lt;/p&gt;</comment>
                            <comment id="1758170" author="bruce.lucas@10gen.com" created="Thu, 21 Dec 2017 20:22:33 +0000"  >&lt;p&gt;Thanks Alex, good to hear confirmation that the tickets you mention will fix this. I&apos;m recording the symptoms I&apos;m seeing here to help in diagnosis pending those fixes.&lt;/p&gt;

&lt;p&gt;I&apos;ll also note that the shutdown process that you describe is extremely memory-intensive. Example:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Heavy insert workload of about 40 MB/s built up a replication lag of about 2.5 min, and resulted in a lookaside table of about 83 M entries with lookaside file of about 3.7 GB on the primary.&lt;/li&gt;
	&lt;li&gt;Let the system quiesce for a few minutes so that the secondary caught up, there were no longer any transactions pinned by timestamps, and all the data had been checkpointed by the periodic checkpoints.&lt;/li&gt;
	&lt;li&gt;Shutdown of mongod then took many minutes while memory usage grew from ~ 8 GB (the configured cache size) to use the full 64 GB of memory on the machine. (Actually I didn&apos;t wait for shutdown to complete because the memory usage had brought the machine to a standstill.)&lt;/li&gt;
&lt;/ul&gt;

</comment>
                            <comment id="1757467" author="alexander.gorrod" created="Thu, 21 Dec 2017 01:02:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; The reason it takes a long time is that while the oldest ID is pinned we need to write data into the lookaside file instead of the data table. When we then do the shutdown, and the ID is no longer pinned that content is read from the lookaside file and written into the data file. There isn&apos;t much we can do to avoid that read. It would be possible to skip that step, and instead recover via the oplog from further back in time, but I expect that would make startup time much longer - so I don&apos;t think it&apos;d be a net win.&lt;/p&gt;

&lt;p&gt;On the other hand the changes in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3767&quot; title=&quot;Avoid lookaside instantiation for faster reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3767&quot;&gt;&lt;del&gt;WT-3767&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3768&quot; title=&quot;Lookaside optimization: birthmarks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3768&quot;&gt;&lt;del&gt;WT-3768&lt;/del&gt;&lt;/a&gt; do have relevant lookaside table optimizations, so I expect the behavior to improve once those changes are done and available in MongoDB.&lt;/p&gt;</comment>
                            <comment id="1757072" author="bruce.lucas@10gen.com" created="Wed, 20 Dec 2017 19:08:20 +0000"  >&lt;p&gt;Here&apos;s an example:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/174101/174101_index-build.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Start an index build of a collection with 1.5 M documents at B on the primary&lt;/li&gt;
	&lt;li&gt;It completes on the primary ~100 seconds later at C, and the index build on the secondary starts&lt;/li&gt;
	&lt;li&gt;This halts replication from C-D, which creates cache pressure due to having to keep uncommitted updates, and generates 3.7 M lookaside table entries.&lt;/li&gt;
	&lt;li&gt;The index build on the secondary completes ~40 seconds later at D, and we begin to catch up.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;A subsequent shutdown of the primary (not shown) took ~45 seconds. In a larger example with a larger lag shutdown took many minutes.&lt;/p&gt;

&lt;p&gt;Stack traces like the following indicating LAS activity during shutdown:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x0000561e818751c4 in bool snappy::InternalUncompress&amp;lt;snappy::SnappyArrayWriter&amp;gt;(snappy::Source*, snappy::SnappyArrayWriter*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x0000561e8187587d in snappy::RawUncompress(char const*, unsigned long, char*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x0000561e81873d3e in snappy_uncompress ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x0000561e8023de55 in snappy_decompression ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x0000561e802489ea in __wt_bt_read ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x0000561e8024fc02 in __wt_page_in_func ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x0000561e802792d4 in __wt_row_search ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x0000561e802d3342 in __wt_btcur_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x0000561e8028bcbd in __curfile_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#9  0x0000561e8027c48e in __wt_las_cursor_position ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#10 0x0000561e80250e72 in __wt_page_in_func ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#11 0x0000561e8026bec0 in __wt_page_swap_func ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#12 0x0000561e8026e02a in __tree_walk_internal ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#13 0x0000561e802a05d7 in __wt_evict_file ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#14 0x0000561e80262518 in __wt_cache_op ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#15 0x0000561e802334f1 in __checkpoint_tree ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#16 0x0000561e80236b7f in __wt_checkpoint_close ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#17 0x0000561e801d7a63 in __wt_conn_dhandle_close ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#18 0x0000561e801d84a7 in __wt_conn_dhandle_discard_single ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#19 0x0000561e801d85e4 in __wt_conn_dhandle_discard ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#20 0x0000561e801d9548 in __wt_connection_close ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#21 0x0000561e801d1a02 in __conn_close ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#22 0x0000561e801a7208 in mongo::WiredTigerKVEngine::cleanShutdown() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#23 0x0000561e8037d3ee in mongo::ServiceContextMongoD::shutdownGlobalStorageEngineCleanly() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#24 0x0000561e8016ed2e in mongo::(anonymous namespace)::shutdownTask() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1757038" author="bruce.lucas@10gen.com" created="Wed, 20 Dec 2017 18:29:07 +0000"  >&lt;p&gt;One consequence of this that I&apos;ve observed is that if there has been substantial lookaside activity, for example because replication lag during a write-heavy workload has created cache pressure due to the requirement to keep uncommited updates in cache on the primary, then normal shutdown of the node can take a very long time related to lookaside table cleanup (I think, based on collected stack traces).&lt;/p&gt;</comment>
                            <comment id="1752809" author="alexander.gorrod" created="Fri, 15 Dec 2017 05:26:16 +0000"  >&lt;p&gt;I&apos;ve reproduced this behavior and have a few observations, but no clear conclusion yet:&lt;/p&gt;

&lt;p&gt;There is no obvious correlation between checkpoints or other periodic activity and lookaside writes. &lt;br/&gt;
There does appear to be a correlation between IO saturation and lookaside writes.&lt;br/&gt;
There is a correlation between page splits and lookaside writes, but it isn&apos;t clear to me whether it&apos;s cause or effect.&lt;br/&gt;
Lookaside records are being removed, but it appears as though not all of them are being cleaned up.&lt;br/&gt;
There is lag in the readConcernMajority corresponding to the times when lookaside is used.&lt;br/&gt;
The cache isn&apos;t at the dirty limit when lookaside writes start happening.&lt;/p&gt;

&lt;p&gt;My suspicion is that this workload overwhelms IO. I&apos;ve been running with the journal and data files on the same drive. I think a journal flush is probably taking a long time due to the disk saturation, which means the oldest timestamp isn&apos;t being updated for several seconds, which in turn leads to lookaside usage as we try to evict pages with history that&apos;s required to satisfy read concern majority.&lt;/p&gt;

&lt;p&gt;I&apos;ve attempted to capture a set of statistics that reflect my above observations:&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/173599/173599_Screen+Shot+2017-12-15+at+4.12.53+pm.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;There are some things I&apos;d like to understand:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Why aren&apos;t isn&apos;t more space being reclaimed from the lookaside file (I suspect it&apos;s because the workload is never reading the data back in, so it&apos;s not being cleaned up. The work in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766&quot; title=&quot;Lookaside sweep for obsolete updates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3766&quot;&gt;&lt;del&gt;WT-3766&lt;/del&gt;&lt;/a&gt; should address that issue.&lt;/li&gt;
	&lt;li&gt;Why is the workload triggering lookaside writes. I don&apos;t think they are helpful here, and I suspect we can do some work to adjust either the types of pages we choose to do lookaside writes of or the &quot;lookaside score&quot; heuristic which is used to decide whether to enable lookaside writes and see this workload run better.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1741151" author="alexander.gorrod" created="Mon, 4 Dec 2017 04:33:15 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; I believe you are noticing an absence of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766&quot; title=&quot;Lookaside sweep for obsolete updates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3766&quot;&gt;&lt;del&gt;WT-3766&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The way lookaside (cache overflow) works is that if we need to reclaim space from cache, we write that content into the lookaside file. The content in the lookaside file is then removed the next time that data is accessed or when a table is dropped. It&apos;s not surprising to me that a write only workload will show lookaside file growth, since there are no queries causing WiredTiger to remove redundant content from the lookaside file. Implementing an alternative mechanism for reclaiming that space is the goal of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3766&quot; title=&quot;Lookaside sweep for obsolete updates&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3766&quot;&gt;&lt;del&gt;WT-3766&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I believe this particular workload could use the lookaside file less - but that&apos;s likely going to be covered by work in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32139&quot; title=&quot;Oplog truncation creates large amount of dirty data in cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32139&quot;&gt;&lt;del&gt;SERVER-32139&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="463678">WT-3766</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="496646">WT-3911</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="502214">WT-3937</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="467003">SERVER-32139</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="463678">WT-3766</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="173599" name="Screen Shot 2017-12-15 at 4.12.53 pm.png" size="472889" author="alexander.gorrod@mongodb.com" created="Fri, 15 Dec 2017 05:19:11 +0000"/>
                            <attachment id="172231" name="dd-rc8-small.tar" size="702976" author="bruce.lucas@mongodb.com" created="Sun, 3 Dec 2017 15:09:53 +0000"/>
                            <attachment id="172228" name="growth.png" size="149142" author="bruce.lucas@mongodb.com" created="Sun, 3 Dec 2017 14:50:39 +0000"/>
                            <attachment id="174101" name="index-build.png" size="165522" author="bruce.lucas@mongodb.com" created="Wed, 20 Dec 2017 19:06:40 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000Zsg6lIAB, 500A000000bTCFQIA4]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 3 Dec 2017 22:44:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 27 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WT-3911'>WT-3911</a></s>, <s><a href='https://jira.mongodb.org/browse/WT-3766'>WT-3766</a></s>, <s><a href='https://jira.mongodb.org/browse/WT-3937'>WT-3937</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 27 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htlh6f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8quv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htl3af:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>