<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:17:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-48522] Regression after mass deletion</title>
                <link>https://jira.mongodb.org/browse/SERVER-48522</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Test inserts 100 k documents, removes them all, then looks for (the now non-existent) documents via the _id index.&lt;/p&gt;

&lt;p&gt;Left is 4.2.6, right is 4.4.0-rc7&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/263553/263553_regression.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;A-B, D-E: removing 100 k documents. There is a regression of about 30% in rate of deletion (&quot;cursor remove calls&quot;)&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;starting B, E: doing a find via the _id index for a non-existent document&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;B-C: in 4.2.6 for a period of time the queries are slow, then at C the rate picks up, apparently having something to do with pages evicted due to too many deleted items&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;E-F: in 4.4.0-rc7 the rate is also intially slow (albeit slower than in 4.2.6 at the same point), then picks up some at F, but remains much slower than in 4.2.6. Behavior of pages evicted due to too many deleted items is different - looks like we&apos;re trying but not able to evict them?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;PMP shows most of our time here&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00005652ac555e4c in __wt_value_return_buf ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00005652ac5248b7 in __wt_txn_read ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00005652ac525bdf in __wt_btcur_next ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00005652ac53294a in __wt_btcur_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x00005652ac484528 in __curfile_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x00005652ac41695f in int mongo::wiredTigerPrepareConflictRetry&amp;lt;mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;)::{lambda()#1}&amp;gt;(mongo::OperationContext*, mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;)::{lambda()#1}&amp;amp;&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x00005652ac417070 in mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x00005652ac41d212 in mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seek(mongo::KeyString::Value const&amp;amp;, mongo::SortedDataInterface::Cursor::RequestedInfo) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x00005652ace06b22 in mongo::IndexScan::initIndexScan() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#9  0x00005652ace07817 in mongo::IndexScan::doWork(unsigned long*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This one also showed up once&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00005652ac52331e in __wt_txn_upd_value_visible_all ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00005652ac526917 in __wt_btcur_next ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00005652ac53294a in __wt_btcur_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00005652ac484528 in __curfile_search_near ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x00005652ac41695f in int mongo::wiredTigerPrepareConflictRetry&amp;lt;mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;)::{lambda()#1}&amp;gt;(mongo::OperationContext*, mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;)::{lambda()#1}&amp;amp;&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x00005652ac417070 in mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seekWTCursor(mongo::KeyString::Value const&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x00005652ac41d212 in mongo::(anonymous namespace)::WiredTigerIndexCursorBase::seek(mongo::KeyString::Value const&amp;amp;, mongo::SortedDataInterface::Cursor::RequestedInfo) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x00005652ace06b22 in mongo::IndexScan::initIndexScan() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x00005652ace07817 in mongo::IndexScan::doWork(unsigned long*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Repro: 3-node replica set&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;function repro() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    // insert 100 k documents&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    nthreads = 10&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    threads = []&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (var t=0; t&amp;lt;nthreads; t++) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        thread = new ScopedThread(function(t) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            size = 10&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            count = 10*1000;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            every = 1000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            const x = &apos;x&apos;.repeat(size)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            for (var i=0; i&amp;lt;count; i+=every) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                many = []&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                for (var j=0; j&amp;lt;every; j++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                    many.push({_id: t*count+i+j, x})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                db.c.insertMany(many)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                if (i%10000==0) print(t, i)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }, t)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        threads.push(thread)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        thread.start()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (var t = 0; t &amp;lt; nthreads; t++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        threads[t].join()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    // remove them&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    print(&quot;removing&quot;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    db.c.remove({_id: {$lt: 100000}})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    // loop trying to find non-existent document via _id index&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    print(&quot;finding&quot;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    while (true)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        db.c.find({_id: {$lt: 100000}}).itcount()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

</description>
                <environment></environment>
        <key id="1368186">SERVER-48522</key>
            <summary>Regression after mass deletion</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="1" iconUrl="https://jira.mongodb.org/images/icons/statuses/open.png" description="">Open</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="siddhartha.mahajan@mongodb.com">Sid Mahajan</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>KP44</label>
                            <label>dmd-escapes</label>
                            <label>perf-escapes</label>
                            <label>refinement</label>
                    </labels>
                <created>Mon, 1 Jun 2020 18:39:01 +0000</created>
                <updated>Fri, 28 Apr 2023 13:58:29 +0000</updated>
                                            <version>4.4.0-rc7</version>
                                                                        <votes>0</votes>
                                    <watches>42</watches>
                                                                                                                <comments>
                            <comment id="5157677" author="JIRAUSER1257956" created="Wed, 1 Feb 2023 04:40:24 +0000"  >&lt;p&gt;Fixing the root cause found for the &lt;a href=&quot;https://jira.mongodb.org/browse/WT-10424&quot; title=&quot;cursor::search_near slow performance if many deleted items are present &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-10424&quot;&gt;&lt;del&gt;WT-10424&lt;/del&gt;&lt;/a&gt; issue also fixes this issue.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Summary of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-10424&quot; title=&quot;cursor::search_near slow performance if many deleted items are present &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-10424&quot;&gt;&lt;del&gt;WT-10424&lt;/del&gt;&lt;/a&gt;&lt;/b&gt; (detailed info on the ticket)&lt;b&gt;:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;The ticket concerns the slow search_near performance that is caused by a high number of deleted items and large cache size. Lowering the cache size was found to greatly improve the performance, but further investigation uncovered that the pages with many deleted items were not being evicted. The issue was that these pages were not being added to the urgent eviction queue as they were already in the ordinary queues, however, the eviction threads found the ordinary queues empty. This led to the persistence of deleted pages in the cache, hindering the search_near performance.&lt;/p&gt;

&lt;p&gt;To resolve the problem, the solution is to remove the relevant (non-candidate) pages from the queue, thereby improving the search_near performance.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FTDC analysis before and after &lt;a href=&quot;https://jira.mongodb.org/browse/WT-10424&quot; title=&quot;cursor::search_near slow performance if many deleted items are present &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-10424&quot;&gt;&lt;del&gt;WT-10424&lt;/del&gt;&lt;/a&gt; changes:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Before:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/429292/429292_Screen+Shot+2023-02-01+at+3.31.12+pm.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;After:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/429293/429293_Screen+Shot+2023-02-01+at+3.31.00+pm.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Thanks, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=etienne.petrel%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;etienne.petrel@mongodb.com&quot;&gt;etienne.petrel@mongodb.com&lt;/a&gt; for running the tests. The &lt;tt&gt;opcounters query&lt;/tt&gt; rate has noticeably increased after the implementation of the changes in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-10424&quot; title=&quot;cursor::search_near slow performance if many deleted items are present &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-10424&quot;&gt;&lt;del&gt;WT-10424&lt;/del&gt;&lt;/a&gt;. As the root cause is similar, this ticket will be marked as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-10424&quot; title=&quot;cursor::search_near slow performance if many deleted items are present &quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-10424&quot;&gt;&lt;del&gt;WT-10424&lt;/del&gt;&lt;/a&gt;. However, if any future regressions arise, feel free to re-open this ticket for further investigation.&lt;/p&gt;</comment>
                            <comment id="4411918" author="JIRAUSER1263295" created="Tue, 15 Mar 2022 02:20:57 +0000"  >&lt;p&gt;&amp;gt; Why are you making the assumption that WT is not the cause?&lt;/p&gt;

&lt;p&gt;I only assumed this for the find regression, since your earlier bisecting showed that came from MDB commit `06a03bc6ec`.&lt;/p&gt;

&lt;p&gt;&amp;gt; Did you do this?&lt;/p&gt;

&lt;p&gt;Yep, hence the comment that there&apos;s still a large regression here - the same 32% (roughly) that was originally observed, so it&apos;s definitely something to do with exhaust cursors.&lt;/p&gt;

&lt;p&gt;&amp;gt; Back in Oct-2021 Alexander Gorrod moved this from replication to storage engines because:&lt;/p&gt;

&lt;p&gt;I think there&apos;s some misunderstanding around what &quot;this&quot; is. My comment on March 8 refers solely to the find regression, but Alex moved the delete regression since that&apos;s definitely a WT issue (due to the addition of durable history).&lt;/p&gt;</comment>
                            <comment id="4397101" author="JIRAUSER1263295" created="Tue, 8 Mar 2022 03:53:26 +0000"  >&lt;p&gt;An update on the read performance issue. Even with very recent MDB + WT, there&apos;s still a large regression here. Based on the above discussion, I started investigating with the assumption that WT was not the cause, so I went looking through MDB server code for what actually changes when exhaust cursors are turned on.&lt;/p&gt;

&lt;p&gt;I don&apos;t know the server code at all, so I limited it to looking at all the places that mention exhaust cursors, but there were still a lot of these - I found three places where the performance hit might be happening.&lt;/p&gt;

&lt;p&gt;The first is in bgsync.cpp, where there&apos;s an optimisation to avoid writing ops one at a time. I&apos;m not sure if this applies for the regression we&apos;re looking at, but it&apos;s interesting that exhaust cursors are excluded. This might be fine, I&apos;m just pointing it out since the motivation there isn&apos;t obvious to me:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// Check some things periodically (whenever we run out of items in the current cursor batch).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;if&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; (!oplogFetcherUsesExhaust &amp;amp;&amp;amp; info.networkDocumentBytes &amp;gt; &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; &amp;amp;&amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        info.networkDocumentBytes &amp;lt; kSmallBatchLimitBytes) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// On a very low latency network, if we don&apos;t wait a little, we&apos;ll be&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// getting ops to write almost one at a time.  This will both be expensive&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// for the upstream server as well as potentially defeating our parallel&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// application of batches on the secondary.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;//&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        &lt;/span&gt;&lt;span style=&quot;color: #008200; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// The inference here is basically if the batch is really small, we are &quot;caught up&quot;.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        sleepmillis(kSleepToAllowBatchingMillis);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The second is in strategy.cpp, where we have some code that parses a reply and creates a response. Again, this might be really cheap but it&apos;s extra work that isn&apos;t being done for non-exhaust cursors. If I knew the server code better I might be able to exclude this.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;if&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; (OpMsg::isFlagSet(m, OpMsg::kExhaustSupported)) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    auto responseObj = reply-&amp;gt;getBodyBuilder().asTempObj();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;if&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; (responseObj.getField(&lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;ok&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;).trueValue()) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        dbResponse.shouldRunAgainForExhaust = reply-&amp;gt;shouldRunAgainForExhaust();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        dbResponse.nextInvocation = reply-&amp;gt;getNextInvocation();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&#160;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Finally, service_state_machine.cpp has a `makeExhaustMessage` function that seems to do quite a lot of work, including `OpMsg::appendChecksum(&amp;amp;exhaustMessage)`, which is probably expensive for longer messages. It&apos;s unclear how often this is called though.&lt;/p&gt;</comment>
                            <comment id="4379387" author="JIRAUSER1263295" created="Mon, 28 Feb 2022 00:14:08 +0000"  >&lt;p&gt;I&apos;m picking this up after Monica and Alexey had to move on to something else, so here&apos;s my understanding of the situation to make sure I&apos;m up to speed.&lt;/p&gt;

&lt;p&gt;There are two perf regressions here, both happened at around the same time. One turned out to be MDB (the &quot;find&quot; regression), another turned out to be WT (the &quot;remove&quot; regression). The &quot;find&quot; regression went away when testing WT4.4 with MDB 4.2, but the remove regression didn&apos;t.&lt;/p&gt;

&lt;p&gt;One thing that I don&apos;t think has been explained yet is that, for this workload, we kind of expect to see the &quot;remove&quot; regression. The WT version that MDB 4.4 uses introduced durable history, and we&apos;re hitting something of a pathological case here with the &quot;insert then immediately remove&quot; pattern. After durable history, we keep these records in the data store for a few seconds, so now when we search for the non-existent entry, we actually have to do some work - we can&apos;t &quot;short circuit&quot; the search like we did previously. (I think this &quot;few seconds&quot; might be changing to &quot;30 seconds&quot; in an upcoming WT version, but don&apos;t quote me on that).&lt;/p&gt;

&lt;p&gt;That said, this is just an explanation of what&apos;s happening; it might have changed since &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7757&quot; title=&quot;Skip obsolete leaf pages without having to read them&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7757&quot;&gt;&lt;del&gt;WT-7757&lt;/del&gt;&lt;/a&gt; where we did some general optimisations in this area. This is a somewhat strange workload and not something we expect to hit in the real world, but the oplog is a little bit special and is hitting it.&lt;/p&gt;

&lt;p&gt;Ger bisected the &quot;find&quot; regression to the MDB commit that started using exhaust cursors - I suspect that one&apos;s out of WT&apos;s hands since it&apos;s asking us to do more stuff. Confirming that is the next step for me - I plan to take a look at exactly what changes when using the exhaust cursor.&lt;/p&gt;</comment>
                            <comment id="4361260" author="bruce.lucas@10gen.com" created="Thu, 17 Feb 2022 14:09:02 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=monica.ng&quot; class=&quot;user-hover&quot; rel=&quot;monica.ng&quot;&gt;monica.ng&lt;/a&gt; if I&apos;m not mistaken ScopedThread was renamed to Thread somewhere along the way.&lt;/p&gt;</comment>
                            <comment id="4360701" author="JIRAUSER1257876" created="Thu, 17 Feb 2022 03:52:38 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat&quot;&gt;dmitry.agranat&lt;/a&gt; The original reproducer uses the ScopedThread class but this is not supported in the MongoDB shell anymore, is there any equivalent class we can use instead? I have a reproducer script in Python but I&apos;m not sure it is doing exactly the same thing.&lt;/p&gt;</comment>
                            <comment id="4345277" author="lingzhi.deng" created="Wed, 9 Feb 2022 15:59:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=monica.ng&quot; class=&quot;user-hover&quot; rel=&quot;monica.ng&quot;&gt;monica.ng&lt;/a&gt; I haven&apos;t conducted the same investigation here. But yes, all you need to do is to turn &lt;a href=&quot;https://docs.mongodb.com/manual/reference/parameters/#mongodb-parameter-param.oplogFetcherUsesExhaust&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;oplogFetcherUsesExhaust&lt;/tt&gt;&lt;/a&gt; off.&lt;/p&gt;</comment>
                            <comment id="4344612" author="JIRAUSER1257876" created="Wed, 9 Feb 2022 12:04:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lingzhi.deng&quot; class=&quot;user-hover&quot; rel=&quot;lingzhi.deng&quot;&gt;lingzhi.deng&lt;/a&gt; I saw your comment here: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48522?focusedCommentId=3700654&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-3700654&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-48522?focusedCommentId=3700654&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-3700654&lt;/a&gt; and wasn&apos;t sure if any investigation was conducted into this.  &lt;/p&gt;

&lt;p&gt;If not, I would like to go down the path of examining the performance trends without the use of exhaust cursors for oplog fetching so that I can potentially isolate any regressions related to WiredTiger specifically. Is there a way to disable the use of exhaust cursors for oplog fetching? Can it be turned off by setting the &quot;oplogFetcherUsesExhaust&quot; parameter? Is there anything else I should be aware of?&lt;/p&gt;</comment>
                            <comment id="4338313" author="JIRAUSER1257876" created="Mon, 7 Feb 2022 00:54:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; I&apos;ve tried running the reproducer against the recent version of MongoDB which would include WiredTiger changes like &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7757&quot; title=&quot;Skip obsolete leaf pages without having to read them&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7757&quot;&gt;&lt;del&gt;WT-7757&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6707&quot; title=&quot;Allow eviction to operate with snapshot when checkpoint is running&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6707&quot;&gt;&lt;del&gt;WT-6707&lt;/del&gt;&lt;/a&gt;. We thought these may help to improve performance around the query rate, but it doesn&apos;t seem like it does. From what I read above, it also looks like the reason for the regression could be coming from the MongoDB level, I am thinking of passing the ticket back unless you have other thoughts on other things we could look into?&lt;/p&gt;</comment>
                            <comment id="4320218" author="JIRAUSER1257876" created="Fri, 28 Jan 2022 03:51:47 +0000"  >&lt;p&gt;I had a look into this and wanted to check if we were seeing similar trends in the metrics when running the reproducer against a newer version of MongoDB. I am still seeing a drop in the &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ss opcounters query&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; metric although the value of the &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ss wt cursor cursor remove calls&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; remains in the same range.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/357449/357449_Stats.png&quot; width=&quot;650&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We noticed that the &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ss wt cache checkpoint blocked page eviction&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; metric is considerably higher in v5.1.0. I will try to investigate whether reducing the concurrency of checkpoint and eviction will ease at least some of the performance regression. &lt;/p&gt;

</comment>
                            <comment id="4122968" author="alexander.gorrod" created="Wed, 13 Oct 2021 23:01:24 +0000"  >&lt;p&gt;Storage Engines team is going to pick this up and run the reproducer against the latest version of WiredTiger. We did some work in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7757&quot; title=&quot;Skip obsolete leaf pages without having to read them&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7757&quot;&gt;&lt;del&gt;WT-7757&lt;/del&gt;&lt;/a&gt; to skip tombstones more affordably, which will hopefully significantly reduce the performance difference between releases.&lt;/p&gt;

&lt;p&gt;If there is still an unreasonable performance regression, we will do some analysis to find the right owner.&lt;/p&gt;</comment>
                            <comment id="3528753" author="thomas.schubert" created="Mon, 14 Dec 2020 18:28:35 +0000"  >&lt;p&gt;Discussing with Bruce, I think that this would be a good ticket for the performance team to pick up to write a regression test and track down the remaining regression.&lt;/p&gt;</comment>
                            <comment id="3205123" author="bruce.lucas@10gen.com" created="Thu, 11 Jun 2020 14:35:05 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sulabh.mahajan&quot; class=&quot;user-hover&quot; rel=&quot;sulabh.mahajan&quot;&gt;sulabh.mahajan&lt;/a&gt;. I&apos;ve confirmed that, using the same shell in both cases, rc9 and 4.2.6 now show the same performance on the finds.&lt;/p&gt;

&lt;p&gt;However I&apos;m still seeing a regression in the removes. However it looks like WT is being asked to do more work for the same number of documents removed. Left is 4.2.6 and right is 4.4.0-rc9:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/265727/265727_removes.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&quot;cursor remove calls&quot; is the actual work being done, and we&apos;re doing it at a slower rate in 4.4.&lt;/p&gt;

&lt;p&gt;But in addition to the actual work, we are doing a lot more cursor searches, wt transactions, wt set timestamps, etc., even though we are doing the actual work at a slower rate. As you suggest this means something has changed in the upper layers, but I&apos;m not sure at this point whether it is a change in the integration layer or in the query layer, so I&apos;ll investigate some more before reassigning.&lt;/p&gt;</comment>
                            <comment id="3202388" author="sulabh.mahajan" created="Wed, 10 Jun 2020 05:15:17 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, my conclusion is that the latest 4.4 (rc8) shows much lesser regression than the initially reported values. Seems like &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6306&quot; title=&quot;Do not force evict HS pages because of deleted items&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6306&quot;&gt;&lt;del&gt;WT-6306&lt;/del&gt;&lt;/a&gt; (and other work) has helped.&lt;br/&gt;
The remaining regression seems to be coming from layers above WiredTiger. I am passing the ticket back to you for the next steps.&lt;/p&gt;</comment>
                            <comment id="3202382" author="sulabh.mahajan" created="Wed, 10 Jun 2020 04:44:40 +0000"  >&lt;p&gt;Update:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;I spent a lot of time looking at various t2 stats, and could not find anything that made sense wrt to slower query rates&lt;/li&gt;
	&lt;li&gt;I collected perf traces and compared between 4.2.6 and 4.4.0, and there did not seem to be a smoking gun. Wiredtiger also represented a very small portion of the CPU spent.&lt;/li&gt;
	&lt;li&gt;I thought maybe the bottleneck is above WiredTiger, so I decided to use &lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/tree/mongodb-4.2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WiredTiger from 4.2&lt;/a&gt; with MongoDB v4.4 and got interesting results&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;426 = MongoDB 4.2.6&lt;br/&gt;
440 = MongoDB v4.4 (2a20c87a57)&lt;br/&gt;
440WT42 = MongoDB v4.4 (2a20c87a57) with WiredTiger MongoDB-4.2 (8f7554c852)&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/265478/265478_removals-mixed.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;All three versions show similar remove calls, with a slightly higher peak for 4.4.0. This could be because of the short burst nature of the removals, and possibly &lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/265477/265477_query-mixed.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;4.2.6 gets a consistent query rate of 2800 op/s&lt;br/&gt;
4.4.0 gets an initial query rate of 2800 op/s that falls to a consistent rate of 1900 op/s&lt;br/&gt;
4.4.0 with WT from 4.2 gets a consistent query rate 1900 op/s&lt;/p&gt;

&lt;p&gt;I think there is something in MongoDB outside WiredTiger that might be restricting queries at 1900 op/s. I tried to look for reasons inside WiredTiger that could explain a drop from 2800 op/s to 1900 op/s for 4.4.0 but couldn&apos;t locate any.&lt;/p&gt;

</comment>
                            <comment id="3196562" author="sulabh.mahajan" created="Tue, 9 Jun 2020 06:29:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dmitry.agranat&quot; class=&quot;user-hover&quot; rel=&quot;dmitry.agranat&quot;&gt;dmitry.agranat&lt;/a&gt;, I have not identified any ticket in specific. &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6306&quot; title=&quot;Do not force evict HS pages because of deleted items&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6306&quot;&gt;&lt;del&gt;WT-6306&lt;/del&gt;&lt;/a&gt; is already in rc8, which is the one I think got us a large chunk of performance back. I am still looking into why we have slow queries in the latest 4.4.0 code.&lt;/p&gt;</comment>
                            <comment id="3193679" author="bruce.lucas@10gen.com" created="Sat, 6 Jun 2020 13:29:10 +0000"  >&lt;p&gt;I&apos;m not sure - sounds promising, but I don&apos;t think the fixes you mentioned are in an rc yet. Here&apos;s what I&apos;m seeing on (left to right) 4.2.6, rc7, rc8:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/265057/265057_comp-3.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;So on rc8 the query rate is much improved but still considerably lower than 4.2.6, and the remove rate still shows the same regression.  I&apos;ll check again on rc9.&lt;/p&gt;</comment>
                            <comment id="3191720" author="sulabh.mahajan" created="Fri, 5 Jun 2020 06:36:02 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, this ticket reduces to focusing on why the query rate is lower in 4.4.0. Do you agree? Other parameters seem comparable to me.&lt;/p&gt;</comment>
                            <comment id="3189144" author="sulabh.mahajan" created="Thu, 4 Jun 2020 06:51:46 +0000"  >&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/264266/264266_compare.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I ran the reproducer script on a 3 node replica set, on:&lt;br/&gt;
4.2.6;&lt;br/&gt;
4.4.0 (v4.4 @ b7554bc8dc)&lt;br/&gt;
4.4.0-lat (v4.4 @ b7554bc8dc + develop @912c4844f)&lt;/p&gt;

&lt;p&gt;The eviction state is much better, with reduced forced eviction because of the deleted count. This is likely fixed from &lt;a href=&quot;https://jira.mongodb.org/browse/WT-6306&quot; title=&quot;Do not force evict HS pages because of deleted items&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-6306&quot;&gt;&lt;del&gt;WT-6306&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/264267/264267_remove.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;The cursor remove calls appear to be comparable to 4.2.6&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/264268/264268_focused.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;On zooming closer, here are some observations:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;In 4.2.6 the query quickly ramps up to 2880 ops&lt;/li&gt;
	&lt;li&gt;In 4.4.0 it ramps up and then stays at roughly 1980 ops&lt;/li&gt;
	&lt;li&gt;With latest develop query runs at 2500+ ops then falls to 2000 ops&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3175666" author="bruce.lucas@10gen.com" created="Tue, 2 Jun 2020 12:44:44 +0000"  >&lt;p&gt;Couple of observations:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The regression does not manifest for standalone nodes, only for replica sets&lt;/li&gt;
	&lt;li&gt;Setting maxTargetSnapshotHistoryWindowInSeconds=0 eliminates the regression&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;So it appears this has something to do with the handling of timestamps.&lt;/p&gt;</comment>
                            <comment id="3175293" author="brian.lane" created="Tue, 2 Jun 2020 05:53:50 +0000"  >&lt;p&gt;Hey &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; - yeah, I set it to 4.4.0 and we discussed in planning.  We linked a couple of related issues wanting in first then will bring this one in.  I put it at that top of the backlog.&lt;/p&gt;</comment>
                            <comment id="3175277" author="alexander.gorrod" created="Tue, 2 Jun 2020 05:14:03 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=brian.lane&quot; class=&quot;user-hover&quot; rel=&quot;brian.lane&quot;&gt;brian.lane&lt;/a&gt; did you see this ticket? We should get it prioritized when we can.&lt;/p&gt;</comment>
                            <comment id="3152973" author="bruce.lucas@10gen.com" created="Mon, 1 Jun 2020 19:00:07 +0000"  >&lt;p&gt;Thanks for the confirmation.&lt;/p&gt;</comment>
                            <comment id="3152932" author="louis.williams" created="Mon, 1 Jun 2020 18:59:07 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40167&quot; title=&quot;Index key removal should not encounter prepare conflicts on unrelated keys&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40167&quot;&gt;&lt;del&gt;SERVER-40167&lt;/del&gt;&lt;/a&gt; only applies to reads concurrent with prepared transactions. Every WT read operation is wrapped in a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/912d0ead151bc3a5cac261f07c8453a35a73ad10/src/mongo/db/storage/wiredtiger/wiredtiger_prepare_conflict.h#L59-L66&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;wiredTigerPrepareConflictRetry&lt;/a&gt; to retry in the face of prepare conflicts, so this function will show up in any stack trace involving a read.&lt;/p&gt;</comment>
                            <comment id="3152304" author="bruce.lucas@10gen.com" created="Mon, 1 Jun 2020 18:44:38 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; I noticed that&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40167&quot; title=&quot;Index key removal should not encounter prepare conflicts on unrelated keys&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40167&quot;&gt;&lt;del&gt;SERVER-40167&lt;/del&gt;&lt;/a&gt; is in the same general vicinity as it involves removal and index keys, and&#160;wiredTigerPrepareConflictRetry appears on the PMP stacks above. I think this is probably something different but can you take a look to rule out that the integration layer is generating extra work for the storage layer in 4.4?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1867214">SERVER-59776</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2217436">WT-10424</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1234758">WT-5769</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1367545">WT-6349</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1638751">SERVER-54939</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="429293" name="Screen Shot 2023-02-01 at 3.31.00 pm.png" size="97372" author="siddhartha.mahajan@mongodb.com" created="Wed, 1 Feb 2023 04:32:15 +0000"/>
                            <attachment id="429292" name="Screen Shot 2023-02-01 at 3.31.12 pm.png" size="65360" author="siddhartha.mahajan@mongodb.com" created="Wed, 1 Feb 2023 04:31:55 +0000"/>
                            <attachment id="306995" name="Screenshot 2021-03-29 at 15.18.34.png" size="504493" author="ger.hartnett@mongodb.com" created="Mon, 29 Mar 2021 14:18:46 +0000"/>
                            <attachment id="306997" name="Screenshot 2021-03-29 at 15.19.39.png" size="527297" author="ger.hartnett@mongodb.com" created="Mon, 29 Mar 2021 14:20:04 +0000"/>
                            <attachment id="307255" name="Screenshot 2021-03-30 at 16.25.27.png" size="583170" author="ger.hartnett@mongodb.com" created="Tue, 30 Mar 2021 15:26:44 +0000"/>
                            <attachment id="357449" name="Stats.png" size="265129" author="monica.ng@mongodb.com" created="Fri, 28 Jan 2022 03:51:42 +0000"/>
                            <attachment id="265057" name="comp-3.png" size="117209" author="bruce.lucas@mongodb.com" created="Sat, 6 Jun 2020 13:02:54 +0000"/>
                            <attachment id="264266" name="compare.png" size="141208" author="sulabh.mahajan@mongodb.com" created="Thu, 4 Jun 2020 06:40:13 +0000"/>
                            <attachment id="264268" name="focused.png" size="140982" author="sulabh.mahajan@mongodb.com" created="Thu, 4 Jun 2020 06:50:46 +0000"/>
                            <attachment id="306993" name="metrics.2021-03-26T10-09-27Z-00000" size="98901" author="ger.hartnett@mongodb.com" created="Mon, 29 Mar 2021 14:12:38 +0000"/>
                            <attachment id="306994" name="metrics.2021-03-26T21-18-07Z-00000" size="161461" author="ger.hartnett@mongodb.com" created="Mon, 29 Mar 2021 14:13:54 +0000"/>
                            <attachment id="308076" name="new_oplog_fetcher_exhaust_cursor.png" size="189125" author="lingzhi.deng@mongodb.com" created="Mon, 5 Apr 2021 14:01:50 +0000"/>
                            <attachment id="308077" name="old_oplog_fetcher.png" size="177063" author="lingzhi.deng@mongodb.com" created="Mon, 5 Apr 2021 14:02:11 +0000"/>
                            <attachment id="265477" name="query-mixed.png" size="101600" author="sulabh.mahajan@mongodb.com" created="Wed, 10 Jun 2020 04:23:32 +0000"/>
                            <attachment id="263553" name="regression.png" size="151751" author="bruce.lucas@mongodb.com" created="Mon, 1 Jun 2020 18:38:25 +0000"/>
                            <attachment id="263552" name="regression.tgz" size="192495" author="bruce.lucas@mongodb.com" created="Mon, 1 Jun 2020 18:38:25 +0000"/>
                            <attachment id="265478" name="removals-mixed.png" size="54814" author="sulabh.mahajan@mongodb.com" created="Wed, 10 Jun 2020 04:23:32 +0000"/>
                            <attachment id="264267" name="remove.png" size="22793" author="sulabh.mahajan@mongodb.com" created="Thu, 4 Jun 2020 06:46:59 +0000"/>
                            <attachment id="265727" name="removes.png" size="229535" author="bruce.lucas@mongodb.com" created="Thu, 11 Jun 2020 14:34:07 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>25.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001oHwdMQAS, 5006R00001rbTD2QAM, 5006R00001nLn1wQAC]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 1 Jun 2020 18:59:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 1 week, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/PERF-2075'>PERF-2075</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 1 week, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>brian.lane@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>monica.ng@mongodb.com</customfieldvalue>
            <customfieldvalue>siddhartha.mahajan@mongodb.com</customfieldvalue>
            <customfieldvalue>sulabh.mahajan@mongodb.com</customfieldvalue>
            <customfieldvalue>will.korteland@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxnyon:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0m8fb:zzzi</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5665">Storage - Ra 2022-01-24</customfieldvalue>
    <customfieldvalue id="5668">Storage - Ra 2022-02-07</customfieldvalue>
    <customfieldvalue id="5808">Storage - Ra 2022-02-21</customfieldvalue>
    <customfieldvalue id="5809">Storage - Ra 2022-03-07</customfieldvalue>
    <customfieldvalue id="5976">Storage - Ra 2022-03-21</customfieldvalue>
    <customfieldvalue id="6266">Storage - Ra 2022-06-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxnkxz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>