<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:21:16 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49940] Determine why create index concurrency limit in 4.2.3 might not be working for my benchmarks</title>
                <link>https://jira.mongodb.org/browse/SERVER-49940</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I frequently run two benchmark workloads &#8211; Linkbench and the insert benchmark. Secondary indexes are created for both after there is some data in the indexed table. For Linkbench there is one secondary index while in the insert benchmark I use 3 secondary indexes per collection and usually use 8 collections.&lt;/p&gt;

&lt;p&gt;For the insert benchmark creating the secondary indexes is done concurrently with one client per collection (so 8 in parallel). There is only one secondary index for Linkbench.&#160;&lt;/p&gt;

&lt;p&gt;Creating the secondary index is faster in 4.4 than 4.2 for Linkbench. But for the insert benchmark it is almost 2X faster in 4.2 than 4.4.&lt;/p&gt;

&lt;p&gt;My scripts use &quot;indexed rows/second&quot; rather than time to report performance (so larger == faster) and that is &lt;b&gt;ips&lt;/b&gt; in the data that my test scripts report. There is a lot of &lt;a href=&quot;https://docs.google.com/document/d/1yKpvFpiROYL-CeCnxagJ85a4s-NGr25H7wGOkLSycMA/edit?usp=sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;data here&lt;/a&gt; but I will explain what I see using the results &lt;a href=&quot;https://docs.google.com/document/d/1yKpvFpiROYL-CeCnxagJ85a4s-NGr25H7wGOkLSycMA/edit#heading=h.xpwwx1vqsmxx&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;in this section&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;for IO-bound Linkbench indexed rows/s is 247146 and 271190 for 4.2.8 without and with Snappy compression for the database compared to 113216 and 145950 for 4.4.0rc14 without and with Snappy compression.&lt;/li&gt;
	&lt;li&gt;4.4 uses less CPU than 4.2 &amp;#8211; see the cpupq column which is CPU per indexed row&lt;/li&gt;
	&lt;li&gt;4.4 databases (see dbgb1) are larger than 4.2 which might explain why 4.4 reads more from storage / indexed row than 4.2 (see rkbpq). But this doesn&apos;t explain a 2X perf difference&lt;/li&gt;
	&lt;li&gt;rkbps (storage read KB/s from iostat) is almost 2X larger in 4.2 than 4.4 and that might explain the 2X perf difference.&lt;/li&gt;
	&lt;li&gt;however from a CPU-bound test where the collections are cached and there are no storage reads, 4.2 is still a lot faster (maybe 1.5X) than 4.4. Results &lt;a href=&quot;https://docs.google.com/document/d/1yKpvFpiROYL-CeCnxagJ85a4s-NGr25H7wGOkLSycMA/edit#heading=h.y4t926wmxgro&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;are here&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Some performance results from Linkbench &lt;a href=&quot;https://docs.google.com/document/d/1yKpvFpiROYL-CeCnxagJ85a4s-NGr25H7wGOkLSycMA/edit#heading=h.7yajnxue9dgk&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;are here&lt;/a&gt; and the indexed rows/s rate (ips) is better for 4.4 than 4.2&lt;/p&gt;

&lt;p&gt;Next steps for me:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Figure out why 4.2 can read from storage almost 2X faster despite using the same setup via DSI (c3.8xlarge with 20k EBS IOPs)&lt;/li&gt;
	&lt;li&gt;Re-learn what the limits are for memory used by create index sorts in 4.2 and 4.4&lt;/li&gt;
	&lt;li&gt;upload ftdc&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="1422523">SERVER-49940</key>
            <summary>Determine why create index concurrency limit in 4.2.3 might not be working for my benchmarks</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.callaghan@mongodb.com">Mark Callaghan</assignee>
                                    <reporter username="mark.callaghan@mongodb.com">Mark Callaghan</reporter>
                        <labels>
                    </labels>
                <created>Tue, 28 Jul 2020 00:44:20 +0000</created>
                <updated>Mon, 14 Sep 2020 16:50:57 +0000</updated>
                            <resolved>Mon, 14 Sep 2020 16:50:57 +0000</resolved>
                                    <version>4.2.3</version>
                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="3387821" author="mark.callaghan" created="Fri, 11 Sep 2020 17:47:46 +0000"  >&lt;p&gt;I don&apos;t think there is more to do here. AFAIK there are separate tickets open to update the docs&lt;/p&gt;</comment>
                            <comment id="3311881" author="louis.williams" created="Wed, 29 Jul 2020 13:47:10 +0000"  >&lt;p&gt;My memory has not served me well. In 4.2 we don&apos;t run createIndex commands on the thread pool on primaries (The default value &lt;a href=&quot;https://github.com/mongodb/mongo/blob/c7bb043a38360dddfe983e7d887a768888c4f34d/src/mongo/db/commands/create_indexes.cpp#L729&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; is false). The thread pool is only used on secondaries. We lowered the limit because createIndexes commands on primaries can build several indexes at once, but replicate multiple oplog entries, overwhelming secondaries.&lt;/p&gt;

&lt;p&gt;So the behavior you observed makes sense. If you were running in a replica set configuration, you would likely see significant replication lag due to the limited thread pool resources on the secondaries blocking replication. &lt;/p&gt;</comment>
                            <comment id="3311162" author="mark.callaghan" created="Wed, 29 Jul 2020 01:17:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; - let know if someone on your side wants to continue on this. From what I see, my workloads in 4.2 are not limited to 3 concurrent index builds.&lt;/p&gt;

&lt;p&gt;From looking at vmstat output during the create index with 8 concurrent index builds, 4.2.2 and 4.2.3 look the same. They also take a similar amount of time.&lt;/p&gt;

&lt;p&gt;From 3 sets of stack traces obtained via &lt;a href=&quot;https://poormansprofiler.org&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;PMP&lt;/a&gt; for 4.2.3 it looks like 8 create index commands are running concurrently:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      3 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair&amp;lt;mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 tc_free,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::SortedFileWriter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted&amp;lt;mongo::BSONObj&amp;gt;*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::BSONObj::woCompare(mongo::BSONObj,std::_Deque_iterator&amp;lt;std::pair&amp;lt;mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 __strchr_sse2,mongo::dotted_path_support::extractElementAtPathOrArrayAlongPath(mongo::BSONObj,mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector&amp;lt;char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      3 pread64,__posix_file_read,__wt_block_read_off,__wt_bm_read,__wt_bt_read,__wt_page_in_func,__tree_walk_internal,__wt_btcur_next,__curfile_next,int,mongo::WiredTigerRecordStoreCursorBase::next(),mongo::CollectionScan::doWork(unsigned,mongo::PlanStage::work(unsigned,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted&amp;lt;mongo::BSONObj&amp;gt;*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 tcmalloc::DLL_Prepend(tcmalloc::Span*,,tcmalloc::CentralFreeList::ReleaseToSpans(void*),tcmalloc::CentralFreeList::InsertRange(void*,,tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,,tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::WiredTigerRecoveryUnit::beginUnitOfWork(mongo::OperationContext*),mongo::WriteUnitOfWork::WriteUnitOfWork(mongo::OperationContext*),mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector&amp;lt;char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair&amp;lt;mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 __cursor_row_slot_return,__wt_btcur_next,__curfile_next,int,mongo::WiredTigerRecordStoreCursorBase::next(),mongo::CollectionScan::doWork(unsigned,mongo::PlanStage::work(unsigned,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted&amp;lt;mongo::BSONObj&amp;gt;*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      2 mongo::BSONObj::woCompare(mongo::BSONObj,std::pair&amp;lt;mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::commitBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::dumpInsertsFromBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      2 mongo::BSONObj::woCompare(mongo::BSONObj,std::_Deque_iterator&amp;lt;std::pair&amp;lt;mongo::BSONObj,,void,void,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::commitBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::dumpInsertsFromBulk(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::SortedFileWriter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::sorter::NoLimitSorter&amp;lt;mongo::BSONObj,,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::LockManager::unlock(mongo::LockRequest*),mongo::LockerImpl::_unlockImpl(mongo::FastMapNoAlloc&amp;lt;mongo::ResourceId,,mongo::LockerImpl::unlock(mongo::ResourceId),mongo::LockerImpl::saveLockStateAndUnlock(mongo::Locker::LockSnapshot*),mongo::PlanYieldPolicy::_yieldAllLocks(mongo::OperationContext*,,mongo::PlanYieldPolicy::yieldOrInterrupt(std::function&amp;lt;void,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted&amp;lt;mongo::BSONObj&amp;gt;*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 mongo::BSONObj::getField(mongo::StringData),mongo::BtreeKeyGenerator::_extractNextElement(mongo::BSONObj,mongo::BtreeKeyGenerator::_getKeysWithArray(std::vector&amp;lt;char,mongo::BtreeKeyGenerator::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::getKeys(mongo::BSONObj,mongo::AbstractIndexAccessMethod::BulkBuilderImpl::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insert(mongo::OperationContext*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      1 __wt_page_inmem,__wt_page_in_func,__wt_row_search,__wt_btcur_search_near,__curfile_search_near,int,mongo::WiredTigerRecordStoreCursorBase::restore(),mongo::CollectionScan::doRestoreStateRequiresCollection(),mongo::RequiresCollectionStageBase&amp;lt;mongo::Collection,mongo::PlanExecutorImpl::restoreStateWithoutRetrying(),mongo::PlanYieldPolicy::yieldOrInterrupt(std::function&amp;lt;void,mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted&amp;lt;mongo::BSONObj&amp;gt;*,,mongo::MultiIndexBlock::insertAllDocumentsInCollection(mongo::OperationContext*,,mongo::(anonymous,mongo::ErrmsgCommandDeprecated::run(mongo::OperationContext*,,mongo::BasicCommand::Invocation::run(mongo::OperationContext*,,mongo::(anonymous,mongo::(anonymous,mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*,,mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*,,mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,mongo::transport::ServiceExecutorSynchronous::schedule(std::function&amp;lt;void,mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard,,mongo::ServiceStateMachine::_sourceCallback(mongo::Status),mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard),mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard),std::_Function_handler&amp;lt;void,std::_Function_handler&amp;lt;void,mongo::(anonymous,start_thread,clone&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3310587" author="mark.callaghan" created="Tue, 28 Jul 2020 18:59:15 +0000"  >&lt;p&gt;It will take a few days to get all of the results. But from the first version I tested (rc14) create index is now faster in 4.4 than 4.2 for the insert benchmark once I increased the concurrency limit. So I will keep this open now to figure out why I don&apos;t see the impact of the concurrency limit starting in 4.2.3&lt;/p&gt;</comment>
                            <comment id="3310284" author="louis.williams" created="Tue, 28 Jul 2020 17:08:13 +0000"  >&lt;p&gt;Since this has come up before, index builds in 4.4 should be faster in almost every way due to KeyString sorting improvements we made.  That said, 4.4 introduced some slowness due to the two-phase commit protocol, but unless there&#8217;s a large amount of replication lag, it shouldn&#8217;t be significant at scale.  The observation that index builds are 2x slower seems related only to concurrent behavior when building more indexes than the default limit of 3, but I am interested to see what we find here.&lt;/p&gt;</comment>
                            <comment id="3310252" author="louis.williams" created="Tue, 28 Jul 2020 16:51:50 +0000"  >&lt;p&gt;The maxNumActiveUserIndexBuilds server parameter is hardcoded in 4.2 and not tunable. I filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49948&quot; title=&quot;[4.2] Introduce maxNumActiveUserIndexBuilds setParameter&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49948&quot;&gt;&lt;del&gt;SERVER-49948&lt;/del&gt;&lt;/a&gt; to fix that.&lt;/p&gt;</comment>
                            <comment id="3310201" author="louis.williams" created="Tue, 28 Jul 2020 16:23:03 +0000"  >&lt;p&gt;Yes, actually. Wiredtiger has &quot;read once&quot; cursors that we use for index builds. Enabled by the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/26c5900914fda23a6c5ab85995afbab4eddd4e58/src/mongo/db/catalog/multi_index_block.idl#L36-L43&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;useReadOnceCursorsForIndexBuilds&lt;/a&gt; by default. I did a small writeup in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-37590&quot; title=&quot;Investigate performance implications of using WiredTiger read-once cursors&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-37590&quot;&gt;&lt;del&gt;SERVER-37590&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3310183" author="mark.callaghan" created="Tue, 28 Jul 2020 16:16:14 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; - are there plans to get and use a &quot;don&apos;t cache&quot; flag in WiredTiger so that scans done for create index don&apos;t wipe the cache? We added such an option to MyRocks (MySQL+RocksDB) and used it for full scans done during daily logical backup.&lt;/p&gt;</comment>
                            <comment id="3310169" author="mark.callaghan" created="Tue, 28 Jul 2020 16:08:23 +0000"  >&lt;p&gt;I have more to investigate. I have index build results for all 4.2 releases and I don&apos;t see a change at 4.2.3.&lt;/p&gt;</comment>
                            <comment id="3310061" author="louis.williams" created="Tue, 28 Jul 2020 15:28:45 +0000"  >&lt;p&gt;There is actually more history than I remember:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;A limit of 10 was originally introduced by&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-38323&quot; title=&quot;Create an index builds interface for the embedded version of the server&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-38323&quot;&gt;&lt;del&gt;SERVER-38323&lt;/del&gt;&lt;/a&gt; (for 4.1.7). Undocumented as far as I can tell.&lt;/li&gt;
	&lt;li&gt;The limit was lowered to 3 by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44984&quot; title=&quot;Reduce index thread pool size and reduce memory used per build&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44984&quot;&gt;&lt;del&gt;SERVER-44984&lt;/del&gt;&lt;/a&gt; (for 4.2.3 and 4.4.0-rc0) and the default memory usage was lowered to 200MB. Only the memory usage change was documented by &lt;a href=&quot;https://jira.mongodb.org/browse/DOCS-13432&quot; title=&quot;Investigate changes in SERVER-44984: Reduce index thread pool size&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DOCS-13432&quot;&gt;&lt;del&gt;DOCS-13432&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47155&quot; title=&quot;Limit the number of simultaneous index builds running from user connections to 3&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47155&quot;&gt;&lt;del&gt;SERVER-47155&lt;/del&gt;&lt;/a&gt; changed the implementation to be less buggy and introduced the new parameter that only exists in 4.4.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;In your tests, both 4.2.8 and 4.4.0-rc* should be limited to 3 concurrent index builds, so maybe there is more to investigate there.&lt;/p&gt;</comment>
                            <comment id="3309989" author="louis.williams" created="Tue, 28 Jul 2020 15:01:49 +0000"  >&lt;p&gt;I&apos;m sorry you ran into this issue. I updated the description in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47155&quot; title=&quot;Limit the number of simultaneous index builds running from user connections to 3&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47155&quot;&gt;&lt;del&gt;SERVER-47155&lt;/del&gt;&lt;/a&gt; to provide more context for the change, which was to bound resource utilization of index builds by a safe default.&lt;/p&gt;</comment>
                            <comment id="3309921" author="mark.callaghan" created="Tue, 28 Jul 2020 14:37:21 +0000"  >&lt;p&gt;I was not aware of that change and will repeat my tests. Can server-47155 get updated to explain why this was added?&lt;/p&gt;

&lt;p&gt;Also, I don&apos;t see docs for this:&lt;br/&gt;
&lt;a href=&quot;https://docs.mongodb.com/master/reference/operator/aggregation/lookup/?searchProperty=current&amp;amp;query=maxNumActiveUserIndexBuilds&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/master/reference/operator/aggregation/lookup/?searchProperty=current&amp;amp;query=maxNumActiveUserIndexBuilds&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3309798" author="louis.williams" created="Tue, 28 Jul 2020 14:01:55 +0000"  >&lt;p&gt;In 4.4, we introduced a limit for the maximum number of concurrent index builds to 3 by default (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-47155&quot; title=&quot;Limit the number of simultaneous index builds running from user connections to 3&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-47155&quot;&gt;&lt;del&gt;SERVER-47155&lt;/del&gt;&lt;/a&gt;). Are you raising the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9d420697bd801b8faf0937fb417bc5c58f92872d/src/mongo/db/storage/two_phase_index_build_knobs.idl#L41&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;maxNumActiveUserIndexBuilds&lt;/a&gt; parameter in your test that is running 8 concurrent index builds?&lt;/p&gt;</comment>
                            <comment id="3308636" author="mark.callaghan" created="Tue, 28 Jul 2020 01:43:57 +0000"  >&lt;p&gt;For insert benchmark with 1 collection and serial create index, creating secondary indexes is slightly faster in 4.4 than in 4.2. I will run a few more tests to determine how that changes as concurrency is increased.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 28 Jul 2020 14:01:55 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 21 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 21 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.callaghan@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxx1lb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxjuon:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Create indexes concurrently for large collections&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxwnun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>