<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:46:30 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18079] Large performance drop with documents &gt; 16k on Windows</title>
                <link>https://jira.mongodb.org/browse/SERVER-18079</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The Windows Heap is implemented in user-space as part of kernel32.dll. It divides the heap into two portions:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the Low-Fragmentation Heap (LFH) which is optimized for allocations &amp;lt; 16 KB&lt;/li&gt;
	&lt;li&gt;the large allocation portion which takes a CriticalSection for each allocation and free.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;When WiredTiger needs to make many allocations &amp;gt;= 16KB  performance on Windows will to suffer once the WT cache fills up.  This is easily triggered by workloads with large documents.&lt;br/&gt;
The solution we&apos;re pursuing is to use tc_malloc in place of the system allocator for WT on windows.&lt;br/&gt;
See below for more details and the original bug report.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Original Description&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71075/71075_drop.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;At D performance drops and remains low for the second half of the run (first row).&lt;/li&gt;
	&lt;li&gt;This coincides with the oplog reaching capacity (second row).&lt;/li&gt;
	&lt;li&gt;It also coincides with an increase in heap lock contention (third row), presumably related to the increased memory allocator activity caused by deleting entries from the oplog. Note that heap lock contention has about doubled in spite of the operation rate having fallen by about 4x.&lt;/li&gt;
	&lt;li&gt;A couple of other momentary dips in performance (A, B, C) appear to be related to disk activity (last two rows), and in fact heap lock contention fell during those dips because of the reduced op rate.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Interestingly, the drop is much more pronounced with a 2-node replica set than it is with a 1-node replica set (write concern 1 in both cases). The stack traces confirm the contention and show why a 2-node replica set is worse:&lt;/p&gt;

&lt;p&gt;During the second half of the run there are significant periods when all 10 threads are waiting for the oplog deleter thread to reduce the size of the oplog:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71074/71074_stacks-waiters.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The oplog deleter thread is held up by significant contention in the allocator, in 3 different places:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71072/71072_stacks-deleter.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;A primary source of contention seems to be the getmore that is tailing the oplog, which explains why a 2-node replica set is worse. Note that the oplog tail getmore is having to page in the entries even though they are brand new, which explains why it generates contention, but that seems non-optimal in itself, independent of the allocator issues.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71073/71073_stacks-oplog-tail.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Repro takes about 3 minutes (Windows on VMware instance, 6 cpus, 12 GB memory). Initialize 2-node replica set with these options:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    mongod --storageEngine wiredTiger --wiredTigerCacheSizeGB 5 --oplogSize 6000 ...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Then run 10 threads of the following, which runs twice as long as it takes for the oplog to fill in order to generate a good comparison.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;size = 16000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;w = 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;batch = 10&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;function insert() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    x = &apos;&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (var i=0; i&amp;lt;size; i++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        x = x + &apos;x&apos;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    finish = 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    start = t = ISODate().getTime()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (var i=0; finish==0 || t&amp;lt;finish; ) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        var bulk = db.c.initializeUnorderedBulkOp();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (var j=0; j&amp;lt;batch; j++, i++)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            bulk.insert({x:x})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        bulk.execute({w:w});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        t = ISODate().getTime()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        if (i&amp;gt;0 &amp;amp;&amp;amp; i%1000==0) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            var s = db.getSiblingDB(&apos;local&apos;).oplog.rs.stats(1024*1024)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            print(i, s.size)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            if (finish==0 &amp;amp;&amp;amp; s.size&amp;gt;s.maxSize*0.9) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                finish = start + 2*(t-start)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                print(&apos;finishing in&apos;, finish-t, &apos;ms&apos;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment></environment>
        <key id="198321">SERVER-18079</key>
            <summary>Large performance drop with documents &gt; 16k on Windows</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.benvenuto@mongodb.com">Mark Benvenuto</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>FT</label>
                    </labels>
                <created>Thu, 16 Apr 2015 15:26:21 +0000</created>
                <updated>Thu, 29 Oct 2015 19:14:47 +0000</updated>
                            <resolved>Mon, 4 May 2015 21:26:02 +0000</resolved>
                                    <version>3.0.1</version>
                                    <fixVersion>3.0.4</fixVersion>
                    <fixVersion>3.1.3</fixVersion>
                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="930855" author="xgen-internal-githook" created="Wed, 3 Jun 2015 19:03:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/7e62741800a6e458ab02b9da91e5c37d389c8eae&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/7e62741800a6e458ab02b9da91e5c37d389c8eae&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="930811" author="xgen-internal-githook" created="Wed, 3 Jun 2015 18:25:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 0a2be248888a1c11fda2848682b54fd314ab162c)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b677e49bed78c415498102a6d7d1cfbed43e76f7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b677e49bed78c415498102a6d7d1cfbed43e76f7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="930810" author="xgen-internal-githook" created="Wed, 3 Jun 2015 18:25:50 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 263a5d5b39091a87a553420dba5fb393902a2166)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/7662238b18a136fbad09e45163adff20d32fd1be&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/7662238b18a136fbad09e45163adff20d32fd1be&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="930809" author="xgen-internal-githook" created="Wed, 3 Jun 2015 18:25:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;/p&gt;

&lt;p&gt;(cherry picked from commit b43f9663dd08d3f585a9ae9aa67476b5e1a9d07c)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/99767cdffb16203c5f3190560614b1a166c16bb3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/99767cdffb16203c5f3190560614b1a166c16bb3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="913719" author="sallgeud" created="Thu, 14 May 2015 04:44:59 +0000"  >&lt;p&gt;Is there no performance benefit or value in using TCMalloc for mongo&apos;s heap, too?  &lt;/p&gt;</comment>
                            <comment id="904970" author="xgen-internal-githook" created="Tue, 5 May 2015 04:34:47 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0a2be248888a1c11fda2848682b54fd314ab162c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0a2be248888a1c11fda2848682b54fd314ab162c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="904861" author="xgen-internal-githook" created="Mon, 4 May 2015 23:35:46 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/263a5d5b39091a87a553420dba5fb393902a2166&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/263a5d5b39091a87a553420dba5fb393902a2166&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="904668" author="xgen-internal-githook" created="Mon, 4 May 2015 20:42:42 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18079&quot; title=&quot;Large performance drop with documents &amp;gt; 16k on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18079&quot;&gt;&lt;del&gt;SERVER-18079&lt;/del&gt;&lt;/a&gt;: Use TCMalloc for WT on Windows&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b43f9663dd08d3f585a9ae9aa67476b5e1a9d07c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b43f9663dd08d3f585a9ae9aa67476b5e1a9d07c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="897881" author="mark.benvenuto" created="Fri, 24 Apr 2015 22:00:40 +0000"  >&lt;p&gt;I have run 3 different benchmarks to understand the performance implicaitions of this change:&lt;/p&gt;

&lt;p&gt;Two instance replica set, 5 GB WT cache, 6 GB op log size, same machine (with 32 GB RAM), Windows 8.1 with each replica on separate SSDs.&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;YCSB 4kb documents. This workload is I/O bound on my machine. The spikes on Windows are because of write stalls which enable us to get better read rates. YCSB&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71954/71954_ycsb_4k.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/li&gt;
	&lt;li&gt;YCSB 1kb documents, TC Malloc is a clear win.&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71953/71953_ycsb_1k.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/li&gt;
	&lt;li&gt;SysBench, TCMalloc is a clear win in this case also&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71952/71952_sysbench.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="894400" author="mark.benvenuto" created="Tue, 21 Apr 2015 21:31:34 +0000"  >&lt;p&gt;In answer to earlier inquiries about how does the document size affect the performance, I ran tests with the above repro with 1 KB, 2 KB, 4 KB, and 16 KB documents. The 1 KB documents show little degradation over time. As the sizes of the documents go up, the performance degradation increases, at about 4 KB, we see similar poor performance of the 16 KB document size scenario.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71570/71570_WinHeapVariousDocSizes.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="894265" author="mark.benvenuto" created="Tue, 21 Apr 2015 19:56:47 +0000"  >&lt;p&gt;There is one other option that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=eitan.klein&quot; class=&quot;user-hover&quot; rel=&quot;eitan.klein&quot;&gt;eitan.klein&lt;/a&gt; suggested, use separate Windows heaps for MongoDB and WiredTiger code.&lt;/p&gt;

&lt;p&gt;This approach provides a little benefit as we spend less time fighting over the single lock that guards large allocations: &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71561/71561_DualWinHeap.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This does not provide enough benefit to pursue as I still prefer one of the earlier TCMalloc approaches for WT allocations.&lt;/p&gt;</comment>
                            <comment id="893252" author="mark.benvenuto" created="Mon, 20 Apr 2015 20:20:03 +0000"  >&lt;h3&gt;&lt;a name=&quot;ProblemDescription&quot;&gt;&lt;/a&gt;Problem Description&lt;/h3&gt;

&lt;p&gt;The Windows Heap is implemented in user-space as part of kernel32.dll. It divides the heap into two portions:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;the &lt;a href=&quot;https://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%29.aspx&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Low-Fragmentation Heap (LFH)&lt;/a&gt; which is optimized for allocations &amp;lt; 16 KB&lt;/li&gt;
	&lt;li&gt;the large allocation portion which takes a CriticalSection for each allocation and free.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;WiredTiger does many allocations &amp;gt;= 16KB in Bruce&apos;s workload above which causes performance on Windows to suffer once the WT cache fills up.&lt;/p&gt;

&lt;p&gt;Below is a graph of allocation bucketed by powers of 2 from 2 to 1MB during the workload above. These are only the allocations done by WT, and not the rest of MongoDB. We can see there is a non-zero number of allocations.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/71466/71466_wt_mem_alloc.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;PossibleSolutions&quot;&gt;&lt;/a&gt;Possible Solutions&lt;/h3&gt;

&lt;p&gt;There are two possible solutions:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Do nothing&lt;/li&gt;
	&lt;li&gt;Change the Heap we use MongoDB and/or WiredTiger&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Considering that we already vendor TCMalloc, and use it on Linux, this is the logical heap to use instead of the Windows Heap. There are several different ways to include TCMalloc:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;WT calls tc_malloc functions&lt;/li&gt;
	&lt;li&gt;TCMalloc runtime patches malloc&lt;/li&gt;
	&lt;li&gt;MongoD is statically linked with a libcmt.lib which has no malloc&lt;/li&gt;
&lt;/ol&gt;


&lt;h4&gt;&lt;a name=&quot;1.WTcallstcmalloc&quot;&gt;&lt;/a&gt;1. WT calls tc_malloc&lt;/h4&gt;
&lt;p&gt;Since WT abstracts all allocations calls into os_alloc.c, we could change WT to call tc_malloc instead of malloc. In this choice, allocations made by mongodb code would continue to use TCMalloc, but allocations made by WT would use TCMalloc. This would mean that the process would have two heaps, a Windows heap, and a TCMalloc heap.&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;2.TCMallocruntimepatchesmalloc&quot;&gt;&lt;/a&gt;2. TCMalloc runtime patches malloc&lt;/h4&gt;
&lt;p&gt;Since Windows does not provide an easy way to override the malloc function like Posix Libc, the TCMalloc library will find the addresses for all the various malloc related functions during startup, and replace them with calls to TCMalloc at runtime. This option means we have only heap. The disadvantage is that the TCMalloc disassembler depends on the version of the compiler we use.&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;3.MongoDisstaticallylinkedwithalibcmt.libwhichhasnomalloc&quot;&gt;&lt;/a&gt;3. MongoD is statically linked with a libcmt.lib which has no malloc&lt;/h4&gt;
&lt;p&gt;&lt;a href=&quot;http://src.chromium.org/svn/trunk/src/base/allocator/prep_libc.py&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Chromium&lt;/a&gt; and &lt;a href=&quot;https://dxr.mozilla.org/mozilla-central/source/mozglue/crt/Makefile.in&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Firefox&lt;/a&gt; both do not use the Windows Heap for memory allocations on Windows. Instead, they use TCMalloc, and JEMalloc respectively. They both use different tricks to statically link in their mallocs instead of the CRT&apos;s mallocs. The Chromium method is to create a modified libcmt.lib with malloc removed. The Firefox method is to rely on link order to include jemalloc&apos;s malloc, and modify an object file binary using a python script to do a string replace.&lt;/p&gt;

&lt;p&gt;I do not know if there are any license implications for these two approaches though.&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;Summary&quot;&gt;&lt;/a&gt;Summary&lt;/h4&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Option&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Heap&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Method&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Both&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;tc_ calls&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;TCMalloc&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Runtime patching&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;3&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;TCMalloc&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Static Link&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;My preferences is either for option 1 or 3. There is still work to evaluate the functional and performance effects of these choices. &lt;/p&gt;

&lt;p&gt;CC: &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="200245">WT-1904</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="202817">SERVER-18375</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="198326">SERVER-18081</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="187882">SERVER-17495</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="71561" name="DualWinHeap.png" size="22565" author="mark.benvenuto@mongodb.com" created="Tue, 21 Apr 2015 19:56:28 +0000"/>
                            <attachment id="71649" name="WinHeapBigDocs.png" size="36920" author="mark.benvenuto@mongodb.com" created="Wed, 22 Apr 2015 20:50:53 +0000"/>
                            <attachment id="71570" name="WinHeapVariousDocSizes.png" size="34043" author="mark.benvenuto@mongodb.com" created="Tue, 21 Apr 2015 21:26:17 +0000"/>
                            <attachment id="71075" name="drop.png" size="130262" author="bruce.lucas@mongodb.com" created="Thu, 16 Apr 2015 15:26:21 +0000"/>
                            <attachment id="71072" name="stacks-deleter.png" size="474841" author="bruce.lucas@mongodb.com" created="Thu, 16 Apr 2015 15:26:21 +0000"/>
                            <attachment id="71073" name="stacks-oplog-tail.png" size="220343" author="bruce.lucas@mongodb.com" created="Thu, 16 Apr 2015 15:26:21 +0000"/>
                            <attachment id="71074" name="stacks-waiters.png" size="292561" author="bruce.lucas@mongodb.com" created="Thu, 16 Apr 2015 15:26:21 +0000"/>
                            <attachment id="71952" name="sysbench.png" size="55352" author="mark.benvenuto@mongodb.com" created="Fri, 24 Apr 2015 21:47:42 +0000"/>
                            <attachment id="71466" name="wt_mem_alloc.png" size="231123" author="mark.benvenuto@mongodb.com" created="Mon, 20 Apr 2015 20:14:03 +0000"/>
                            <attachment id="71953" name="ycsb_1k.png" size="41540" author="mark.benvenuto@mongodb.com" created="Fri, 24 Apr 2015 21:47:42 +0000"/>
                            <attachment id="71954" name="ycsb_4k.png" size="52753" author="mark.benvenuto@mongodb.com" created="Fri, 24 Apr 2015 21:47:42 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12451" key="com.atlassian.jira.plugin.system.customfieldtypes:multiversion">
                        <customfieldname>Backport Completed</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="15419">3.0.4</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 20 Apr 2015 20:20:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 37 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>david.daly@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 37 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[Windows]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl82v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs8glj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="501">Platform 3 05/15/15</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11856"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnu27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>