<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:42:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-80124] Measure performance impact of background compaction</title>
                <link>https://jira.mongodb.org/browse/SERVER-80124</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description></description>
                <environment></environment>
        <key id="2418452">SERVER-80124</key>
            <summary>Measure performance impact of background compaction</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="3" iconUrl="https://jira.mongodb.org/images/icons/statuses/inprogress.png" description="This issue is being actively worked on at the moment by the assignee.">In Progress</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="etienne.petrel@mongodb.com">Etienne Petrel</assignee>
                                    <reporter username="gregory.wlodarek@mongodb.com">Gregory Wlodarek</reporter>
                        <labels>
                    </labels>
                <created>Wed, 16 Aug 2023 01:58:16 +0000</created>
                <updated>Tue, 6 Feb 2024 03:21:01 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="6016439" author="JIRAUSER1257954" created="Wed, 17 Jan 2024 06:11:22 +0000"  >&lt;p&gt;Here are a few more interesting stats from &lt;tt&gt;llt_mixed&lt;/tt&gt; that help give context about what compact is doing. &lt;br/&gt;
1. &lt;tt&gt;session table compact passes&lt;/tt&gt; counts the number of &lt;b&gt;passes&lt;/b&gt; compact does over a given table. Compact will do another pass if it has&lt;br/&gt;
	a. Moved some blocks from the last 90%&lt;br/&gt;
	b. Encountered a deleted page.&lt;br/&gt;
	c. Encountered a modified in memory page.&lt;br/&gt;
	In this case &lt;tt&gt;session table compact passes&lt;/tt&gt; reaches ~25 before compact moves on to another table.&lt;br/&gt;
2. &lt;tt&gt;checkpoint progress state&lt;/tt&gt; hovers between 6 (state sync file) and 8 (state bm sync) indicating that there are constant checkpoints happening and could be a bottleneck here.&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/503894/503894_Screenshot+2024-01-17+at+5.05.58%E2%80%AFpm.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;br/&gt;
Zooming in on the final part of the workload it&apos;s evident that compact doesn&apos;t recover any meaningful space until after the workload finishes performing operations and compact is able to move on from the table it was stuck on. Once background compact moves on it quickly passes over all of the other tables and reclaims a significant amount of space. In this case ~15GB. &lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/503897/503897_Screenshot+2024-01-17+at+5.08.16%E2%80%AFpm.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                            <comment id="6012841" author="JIRAUSER1257954" created="Tue, 16 Jan 2024 06:03:35 +0000"  >&lt;p&gt;In comparison to the &lt;tt&gt;ycsb_60GB.long&lt;/tt&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=etienne.petrel%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;etienne.petrel@mongodb.com&quot;&gt;etienne.petrel@mongodb.com&lt;/a&gt; mentions. I would expect &lt;tt&gt;llt_mixed&lt;/tt&gt; to spend less time on compaction since the dataset is much smaller.&lt;/p&gt;

&lt;p&gt;I believe the test operations are somehow causing compact to remain on a single table for an extended period, probably unnecessarily.&lt;/p&gt;</comment>
                            <comment id="5946306" author="JIRAUSER1258229" created="Wed, 13 Dec 2023 00:27:42 +0000"  >&lt;p&gt;After some discussion with the team, we decided to skip that initial checkpoint in the case of background compaction. See &lt;a href=&quot;https://jira.mongodb.org/browse/WT-12132&quot; title=&quot;Make background compaction skip the initial checkpoint in compact&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-12132&quot;&gt;&lt;del&gt;WT-12132&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="5943023" author="JIRAUSER1258229" created="Tue, 12 Dec 2023 04:47:37 +0000"  >&lt;p&gt;I found another test where the performance impact is noticeable &lt;b&gt;while compaction has no work to do&lt;/b&gt;/.&lt;br/&gt;
Patch build: &lt;a href=&quot;https://spruce.mongodb.com/version/65715cc40305b92be166bc79/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;link&lt;/a&gt;&lt;br/&gt;
Base: sys_perf_3ab2f6d2762b0065eae8a857e90d0a2e135f31e4&lt;br/&gt;
Test: &lt;a href=&quot;https://spruce.mongodb.com/task/sys_perf_linux_3_node_replSet_all_feature_flags.2022_11_secondary_performance_patch_3ab2f6d2762b0065eae8a857e90d0a2e135f31e4_65715cc40305b92be166bc79_23_12_07_05_49_25?execution=0&amp;amp;sortBy=STATUS&amp;amp;sortDir=ASC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secondary_performance&lt;/a&gt;&lt;br/&gt;
Perf dashboard: &lt;a href=&quot;https://performance-analyzer.server-tig.prod.corp.mongodb.com/perf-analyzer-viz/?comparison_id=c53cdf0e-11a4-4cbb-b115-cdf49c1ba35b&amp;amp;selected_tab=data-table&amp;amp;determination_filter=Regressed&amp;amp;percent_filter=0%7C%7C100&amp;amp;z_filter=0%7C%7C10&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;link&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In the logs, each compact call results in &lt;tt&gt;there is no useful work to do - skipping compaction&lt;/tt&gt; either because the file size is less than 1MB or the number of available bytes does not meet the default threshold 20MB. So why do we see perf impacts?&lt;/p&gt;

&lt;p&gt;FTDC:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/496262/496262_stats_second_performance.png&quot; width=&quot;100&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;While the server status metrics look quite the same between the baseline and the run with background compaction enabled, we can notice the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;2x times more writes are queued, those are waiting for a ticket or on a lock.&lt;/li&gt;
	&lt;li&gt;There are 55 times checkpoints started by compaction. It is not visible on this screenshot, but the stat that indicates the number of checkpoints triggered by the API is 11 on the baseline.&lt;/li&gt;
	&lt;li&gt;This can be seen as well with the number of files synced by checkpoint, almost 2x more with bg compaction enabled&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Checkpoints seem to be playing a role in the perf impact. Something I have noticed is that we always perform a checkpoint before the two following checks:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The file size is greater than 1MB&lt;/li&gt;
	&lt;li&gt;The number of bytes available is greater than the specified threshold&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Both checks can be impacted by a checkpoint, however, the first one is quite cheap and it could be worth checking this without doing a checkpoint. Even though checkpoint seems more crucial for the second check, I wonder if we could expect the first checkpoint to be done by the application instead of hiding it at the start of compaction.&lt;/p&gt;

&lt;p&gt;I created a &lt;a href=&quot;https://spruce.mongodb.com/version/6577fdfc61837dd639f43e14/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;patch build&lt;/a&gt; where compaction does not generate the first checkpoint, there is no perf regression. I will check with the team if there is something we can do (EDIT: the discussion led to &lt;a href=&quot;https://jira.mongodb.org/browse/WT-12132&quot; title=&quot;Make background compaction skip the initial checkpoint in compact&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-12132&quot;&gt;&lt;del&gt;WT-12132&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;</comment>
                            <comment id="5934920" author="JIRAUSER1258229" created="Fri, 8 Dec 2023 04:03:06 +0000"  >&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/495577/495577_stats_ycsb_60GB_long.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Those are the stats for &lt;tt&gt;ycsb_60GB_long&lt;/tt&gt; base (background compaction disabled) vs bg (background compaction enabled). We can see that eviction is greatly impacted by the background compaction server. Two stats to highlight are:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;cache modified pages evicted by application threads (~x3 with bg compaction)&lt;/li&gt;
	&lt;li&gt;cache pages evicted by application threads (~x2 with bg compaction)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Since application threads are highly solicited to perform eviction, they struggle to maintain the throughput during the test duration, hence the performance regression.&lt;/p&gt;</comment>
                            <comment id="5934821" author="JIRAUSER1258229" created="Fri, 8 Dec 2023 02:22:36 +0000"  >&lt;blockquote&gt;
&lt;p&gt;We are still not able to explain the EBUSY due to a conflicting checkpoint. Why does this specific table take more time than the others that it ends up conflicting with a checkpoint&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;In &lt;a href=&quot;https://spruce.mongodb.com/version/65715cc40305b92be166bc79/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this patch build&lt;/a&gt;, I could observe two things:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;For ycsb_60GB, compaction had nothing to compact and there is no perf regression.&lt;/li&gt;
	&lt;li&gt;For ycsb_60GB.long, compaction found work to do on the collection &lt;tt&gt;collection-41-16287482622083310647.wt&lt;/tt&gt; and this is the timeline:&lt;/li&gt;
&lt;/ul&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# Start of the test&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2023-12-07T08:16:11.794+00:00&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2023-12-07T08:16:13.639+00:00 Compacting file:collection-41-16287482622083310647.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2023-12-07T08:27:03.353+00:00 Compaction conflicts with checkpoint&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# End of the test&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2023-12-07T08:28:37.544+00:00&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Compaction occurred for ~11 min which is the majority of the test duration. It makes sense that compaction had time to conflict with checkpoints as MongoDB tries to take one every minute. The test focuses on this collection while compaction si trying to work on it which creates contention and leads to poor per for the &lt;tt&gt;ycsb_50read50update&lt;/tt&gt; scenario: Z-Score: -80.5, Percent Diff (Region): -58.2%.&lt;/p&gt;</comment>
                            <comment id="5924834" author="JIRAUSER1257954" created="Mon, 4 Dec 2023 23:43:37 +0000"  >&lt;p&gt;This&#160; configuration (&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/494499/494499_background_compact_10gb.txt&quot; title=&quot;background_compact_10gb.txt attached to SERVER-80124&quot;&gt;background_compact_10gb.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;) and change (&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/494500/494500_background_compact.diff&quot; title=&quot;background_compact.diff attached to SERVER-80124&quot;&gt;background_compact.diff&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;) to the background compact cppsuite test readily produces the scenarios of compact completing 100 iterations over a file but failing to recover any space.&#160;&lt;/p&gt;

&lt;p&gt;Some example logs: &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/494501/494501_background_compact_logs.txt&quot; title=&quot;background_compact_logs.txt attached to SERVER-80124&quot;&gt;background_compact_logs.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[1701732616:919716][2044712:0x7fca6c7f8640], file:collection_0.wt, WT_SESSION.compact: [WT_VERB_COMPACT][DEBUG_1]: ============ ending compaction pass&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[1701732616:919753][2044712:0x7fca6c7f8640], file:collection_0.wt, WT_SESSION.compact: [WT_VERB_COMPACT][DEBUG_1]: pages reviewed: 240800&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[1701732616:919769][2044712:0x7fca6c7f8640], file:collection_0.wt, WT_SESSION.compact: [WT_VERB_COMPACT][DEBUG_1]: pages skipped: 240800&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[1701732616:919781][2044712:0x7fca6c7f8640], file:collection_0.wt, WT_SESSION.compact: [WT_VERB_COMPACT][DEBUG_1]: pages rewritten: 0 (297097 expected)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[1701732616:919794][2044712:0x7fca6c7f8640], file:collection_0.wt, WT_SESSION.compact: [WT_VERB_COMPACT][DEBUG_1]: file size 11559MB (12121391104B) with 31% space available 3614MB (3789975552B)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="5922577" author="JIRAUSER1258229" created="Mon, 4 Dec 2023 10:20:21 +0000"  >&lt;p&gt;Update:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;We are still not able to explain the EBUSY due to a conflicting checkpoint. Why does this specific table take more time than the others that it ends up conflicting with a checkpoint&lt;/li&gt;
	&lt;li&gt;We found out that compaction can get stuck or make minimal progress when it finds a page with the &lt;tt&gt;WT_REF_DELETED&lt;/tt&gt; state, waits for a checkpoint to operate on that one, checkpoint does not, compaction finds the same page with &lt;tt&gt;WT_REF_DELETED&lt;/tt&gt;, hopes for a checkpoint to do something, etc. A theory is that we should not expect the following checkpoint to process those pages as it is not guaranteed and force compaction to move on.
	&lt;ul&gt;
		&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/WT-12078&quot; title=&quot;Fix compact to handle pages in WT_REF_DELETED state&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-12078&quot;&gt;&lt;del&gt;WT-12078&lt;/del&gt;&lt;/a&gt; has been created to fix this scenario.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="5911446" author="JIRAUSER1258229" created="Wed, 29 Nov 2023 05:25:15 +0000"  >&lt;p&gt;I have noticed a mistake I made here, I have set up the &lt;tt&gt;free_space_target&lt;/tt&gt; to 1MB which is way too low and will likely trigger compaction as soon as we can recover 1MB. I will re-run a patch with the default value (20MB), it may have no impact in the end but is worth re-trying.&lt;/p&gt;</comment>
                            <comment id="5902227" author="JIRAUSER1258229" created="Fri, 24 Nov 2023 01:07:06 +0000"  >&lt;p&gt;Perf patch builds:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://spruce.mongodb.com/version/655bd3ee61837d8077a9d0c2/tasks&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Patch build 1&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://spruce.mongodb.com/version/655ed50c7742ae45ccf1ddc6/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Patch build 2&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;They unfortunately don&apos;t show good results at first. After investigating with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sean.watt%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;sean.watt@mongodb.com&quot;&gt;sean.watt@mongodb.com&lt;/a&gt;, we have observed an interesting pattern. Here is an example using the &lt;tt&gt;ycsb_60GB.long&lt;/tt&gt; test:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The mongod.0 logs show the following:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:_mdb_catalog.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;_mdb_catalog.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-0-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-0-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-10-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-10-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-12-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-12-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-14-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-14-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-15-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-15-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-17-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-17-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-19-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-19-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-2-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-2-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-22-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-22-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-24-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-24-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-26-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-26-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-27-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-27-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-28-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-28-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-32-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-32-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-35-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-35-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-38-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-38-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&quot;session_dhandle_name&quot;:&quot;file:collection-4-15430505529018804863.wt&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;collection-4-15430505529018804863.wt: there is no useful work to do - skipping compaction&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# 1 minute later&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;t&quot;:{&quot;$date&quot;:&quot;2023-11-23T07:06:28.283+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;WTCMPCT&quot;,  &quot;id&quot;:22430,   &quot;svc&quot;:&quot;u&quot;, &quot;ctx&quot;:&quot;thread12&quot;,&quot;msg&quot;:&quot;WiredTiger message&quot;,&quot;attr&quot;:{&quot;message&quot;:{&quot;ts_sec&quot;:1700723188,&quot;ts_usec&quot;:283501,&quot;thread&quot;:&quot;10970:0xffff805e2d90&quot;,&quot;session_name&quot;:&quot;WT_SESSION.compact&quot;,&quot;category&quot;:&quot;WT_VERB_COMPACT&quot;,&quot;category_id&quot;:9,&quot;verbose_level&quot;:&quot;INFO&quot;,&quot;verbose_level_id&quot;:0,&quot;msg&quot;:&quot;Data handle compaction failed with EBUSY but the cache is not stuck. Will give it another go.&quot;}}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;This most interesting bit is the last line that shows up 40 minutes later. It does not seem right as it means background compaction skipped a bunch of files, then blocked on a specific file to finally get EBUSY.&lt;/li&gt;
	&lt;li&gt;To prove background compaction is running the whole time, we can see on the FTDC that the &lt;tt&gt;table compact running&lt;/tt&gt; is always on which means compaction is active:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/492181/492181_background+compaction+FTDC+perf.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Next steps:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Add more logs to understand what compaction is trying to do&lt;/li&gt;
	&lt;li&gt;Improve the logs to state which dhandle is returning EBUSY 40 min later&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="2418445">SERVER-80121</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2552308">WT-12314</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2547289">WT-12296</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2547290">WT-12297</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2552238">WT-12312</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2553584">WT-12319</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2555141">WT-12328</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2516290">WT-12078</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2523362">WT-12132</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="503450" name="Screenshot 2024-01-16 at 4.45.46&#8239;pm.png" size="186512" author="sean.watt@mongodb.com" created="Tue, 16 Jan 2024 05:47:34 +0000"/>
                            <attachment id="503894" name="Screenshot 2024-01-17 at 5.05.58&#8239;pm.png" size="406599" author="sean.watt@mongodb.com" created="Wed, 17 Jan 2024 06:07:11 +0000"/>
                            <attachment id="503897" name="Screenshot 2024-01-17 at 5.08.16&#8239;pm.png" size="428708" author="sean.watt@mongodb.com" created="Wed, 17 Jan 2024 06:10:23 +0000"/>
                            <attachment id="492181" name="background compaction FTDC perf.png" size="182473" author="etienne.petrel@mongodb.com" created="Fri, 24 Nov 2023 01:06:01 +0000"/>
                            <attachment id="494500" name="background_compact.diff" size="1412" author="sean.watt@mongodb.com" created="Mon, 4 Dec 2023 23:40:39 +0000"/>
                            <attachment id="494499" name="background_compact_10gb.txt" size="975" author="sean.watt@mongodb.com" created="Mon, 4 Dec 2023 23:40:18 +0000"/>
                            <attachment id="494501" name="background_compact_logs.txt" size="35416318" author="sean.watt@mongodb.com" created="Mon, 4 Dec 2023 23:43:39 +0000"/>
                            <attachment id="503895" name="industry_benchmarks_ycsb_95read5update_perf.png" size="316667" author="etienne.petrel@mongodb.com" created="Wed, 17 Jan 2024 06:09:26 +0000"/>
                            <attachment id="503896" name="industry_benchmarks_ycsb_95read5update_t2.png" size="544432" author="etienne.petrel@mongodb.com" created="Wed, 17 Jan 2024 06:09:50 +0000"/>
                            <attachment id="505595" name="linbench2-hs-compact.png" size="89436" author="sean.watt@mongodb.com" created="Tue, 23 Jan 2024 06:41:26 +0000"/>
                            <attachment id="508247" name="linkbench2_disk.png" size="165443" author="etienne.petrel@mongodb.com" created="Wed, 31 Jan 2024 10:32:36 +0000"/>
                            <attachment id="508246" name="linkbench2_latencies.png" size="214391" author="etienne.petrel@mongodb.com" created="Wed, 31 Jan 2024 10:30:45 +0000"/>
                            <attachment id="508245" name="linkbench2_latencies_zoom_out.png" size="208624" author="etienne.petrel@mongodb.com" created="Wed, 31 Jan 2024 10:29:08 +0000"/>
                            <attachment id="508248" name="linkbench2_wiredtiger.png" size="806327" author="etienne.petrel@mongodb.com" created="Wed, 31 Jan 2024 10:35:29 +0000"/>
                            <attachment id="508249" name="linkbench2_wiredtiger_2.png" size="319742" author="etienne.petrel@mongodb.com" created="Wed, 31 Jan 2024 10:37:41 +0000"/>
                            <attachment id="496262" name="stats_second_performance.png" size="687826" author="etienne.petrel@mongodb.com" created="Tue, 12 Dec 2023 04:06:26 +0000"/>
                            <attachment id="495577" name="stats_ycsb_60GB_long.png" size="957492" author="etienne.petrel@mongodb.com" created="Fri, 8 Dec 2023 03:56:27 +0000"/>
                            <attachment id="509528" name="ycsb_60GB.long.2023-09_disk.png" size="270938" author="etienne.petrel@mongodb.com" created="Mon, 5 Feb 2024 04:05:36 +0000"/>
                            <attachment id="509529" name="ycsb_60GB.long.2023-09_latencies.png" size="182506" author="etienne.petrel@mongodb.com" created="Mon, 5 Feb 2024 04:05:36 +0000"/>
                            <attachment id="509530" name="ycsb_60GB.long.2023-09_wiredtiger.png" size="712214" author="etienne.petrel@mongodb.com" created="Mon, 5 Feb 2024 04:05:37 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 24 Nov 2023 01:07:06 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-80121'>SERVER-80121</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3091</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>deepti.hasija@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>etienne.petrel@mongodb.com</customfieldvalue>
            <customfieldvalue>gregory.wlodarek@mongodb.com</customfieldvalue>
            <customfieldvalue>sean.watt@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2mh47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i24j08:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7960">2023-12-12 - Heisenbug</customfieldvalue>
    <customfieldvalue id="7961">2024-01-09 - I Grew Tired</customfieldvalue>
    <customfieldvalue id="8015">StorEng - 2024-01-23</customfieldvalue>
    <customfieldvalue id="8196">2024-02-06 tapioooooooooooooca</customfieldvalue>
    <customfieldvalue id="8197">2024-02-20_A_near-death_puffin</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2m39j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>