<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:42:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16773] Performance degradation due to TCMalloc scalability</title>
                <link>https://jira.mongodb.org/browse/SERVER-16773</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Performance degradation due to TCMalloc scalability with very large number of threads. After a few minutes of running with hundreds of threads, the large majority of CPU time is spent scavenging memory. As this often occurs in critical sections, system throughput may degrade by an order of magnitude or more. Increasing the TCMalloc thread cache to its maximum of 1 GB does not avoid this problem. Using the system allocator does, but costs performance in most other cases.&lt;/p&gt;</description>
                <environment>Mongodb using the TCMalloc memory allocator. </environment>
        <key id="177589">SERVER-16773</key>
            <summary>Performance degradation due to TCMalloc scalability</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="eliot">Eliot Horowitz</assignee>
                                    <reporter username="john.page@mongodb.com">John Page</reporter>
                        <labels>
                    </labels>
                <created>Thu, 8 Jan 2015 12:56:08 +0000</created>
                <updated>Fri, 3 Dec 2021 19:34:36 +0000</updated>
                            <resolved>Mon, 26 Jan 2015 16:34:41 +0000</resolved>
                                    <version>2.8.0-rc4</version>
                                    <fixVersion>3.0.0-rc7</fixVersion>
                                    <component>Performance</component>
                    <component>Storage</component>
                                        <votes>1</votes>
                                    <watches>30</watches>
                                                                                                                <comments>
                            <comment id="1758822" author="wolf_kdy" created="Sat, 23 Dec 2017 07:10:09 +0000"  >&lt;p&gt;Agree with @Igor Canadi&lt;br/&gt;
We also met this situation. &lt;br/&gt;
After about one day&apos;s running with about 4K threads. tcmalloc will somehow rank in front in perf. and mongod costs much more cpu than the normal ones.&lt;br/&gt;
functions about CentralFreeList ranks top.&lt;br/&gt;
After switching to jemalloc,  mongod uses more physical memory, but it doesn&apos;t stale any more.&lt;/p&gt;

&lt;p&gt;Btw, changing tcmalloc&apos;s max_thread_cache offers no help.&lt;br/&gt;
So much memory is left when mongod stales, we find it in serverStatus().&lt;/p&gt;</comment>
                            <comment id="1053058" author="igor" created="Tue, 6 Oct 2015 21:39:04 +0000"  >&lt;p&gt;The same issue was observed by Ceph: &lt;a href=&quot;https://ceph.com/planet/the-ceph-and-tcmalloc-performance-story/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://ceph.com/planet/the-ceph-and-tcmalloc-performance-story/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Switching to jemalloc helped them, too.&lt;/p&gt;</comment>
                            <comment id="1053038" author="igor" created="Tue, 6 Oct 2015 21:26:38 +0000"  >&lt;p&gt;We encountered similar issue with MongoRocks. 35% of the CPU was being spent in tcmalloc, usually in functions related to CentralFreeList (meaning that thread local cache couldn&apos;t fulfill the request). There were a lot of context switches, which indicates lock contention, likely in CentralFreeList. We were running 3.0.6 version with latest Eliot&apos;s commit, meaning that cache size was configured to be 1GB.&lt;/p&gt;

&lt;p&gt;Switching to jemalloc, CPU spent on malloc/free went down to ~4% and latency improved dramatically.&lt;/p&gt;</comment>
                            <comment id="815550" author="xgen-internal-githook" created="Wed, 28 Jan 2015 17:22:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;erh&apos;, u&apos;name&apos;: u&apos;Eliot Horowitz&apos;, u&apos;email&apos;: u&apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: Increase TCMalloc default cache size to 1G&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 9da40029fef37df8d33218101ffa2ff22d94a2da)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/5d71769398636406ef11f850ef6df163a47bb902&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/5d71769398636406ef11f850ef6df163a47bb902&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="815542" author="xgen-internal-githook" created="Wed, 28 Jan 2015 17:17:20 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;erh&apos;, u&apos;name&apos;: u&apos;Eliot Horowitz&apos;, u&apos;email&apos;: u&apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: Increase TCMalloc default cache size to 1G&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9da40029fef37df8d33218101ffa2ff22d94a2da&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9da40029fef37df8d33218101ffa2ff22d94a2da&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813636" author="xgen-internal-githook" created="Mon, 26 Jan 2015 21:53:09 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;GeertBosch&apos;, u&apos;name&apos;: u&apos;Geert Bosch&apos;, u&apos;email&apos;: u&apos;geert@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: Increase TCMalloc default cache size to 256MB&lt;/p&gt;

&lt;p&gt;(cherry picked from commit c30688f704e3fbde4ee83aa2f45a6d79900f10c9)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a0ad9d0380bbe874460151cfa3901b52af28e7c6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a0ad9d0380bbe874460151cfa3901b52af28e7c6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813632" author="xgen-internal-githook" created="Mon, 26 Jan 2015 21:48:50 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;GeertBosch&apos;, u&apos;name&apos;: u&apos;Geert Bosch&apos;, u&apos;email&apos;: u&apos;geert@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: Increase TCMalloc default cache size to 256MB&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c30688f704e3fbde4ee83aa2f45a6d79900f10c9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c30688f704e3fbde4ee83aa2f45a6d79900f10c9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813401" author="xgen-internal-githook" created="Mon, 26 Jan 2015 19:11:39 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;kaloianm&apos;, u&apos;name&apos;: u&apos;Kaloian Manassiev&apos;, u&apos;email&apos;: u&apos;kaloian.manassiev@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt; Get rid of ScopedLock and 2 memory allocations&lt;/p&gt;

&lt;p&gt;This change removes ScopedLock from the RAII lock objects&apos; hierarchy. This&lt;br/&gt;
eliminates two memory allocations and two acquisitions of the Parallel&lt;br/&gt;
Batch Writer mutex.&lt;/p&gt;

&lt;p&gt;I did not see any significant performance improvement, but next change&lt;br/&gt;
would be to remove the allocation of the PBR mutex as well and also to&lt;br/&gt;
make WriteBatchExecutor not allocate lock objects.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit fe3e92d4257b30f01b62d4ef941686b7e0138a8c)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0649555aa3fae26e6770a3186e2766c26fd0cfcf&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0649555aa3fae26e6770a3186e2766c26fd0cfcf&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813224" author="xgen-internal-githook" created="Mon, 26 Jan 2015 17:34:57 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;kaloianm&apos;, u&apos;name&apos;: u&apos;Kaloian Manassiev&apos;, u&apos;email&apos;: u&apos;kaloian.manassiev@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt; Get rid of ScopedLock and 2 memory allocations&lt;/p&gt;

&lt;p&gt;This change removes ScopedLock from the RAII lock objects&apos; hierarchy. This&lt;br/&gt;
eliminates two memory allocations and two acquisitions of the Parallel&lt;br/&gt;
Batch Writer mutex.&lt;/p&gt;

&lt;p&gt;I did not see any significant performance improvement, but next change&lt;br/&gt;
would be to remove the allocation of the PBR mutex as well and also to&lt;br/&gt;
make WriteBatchExecutor not allocate lock objects.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fe3e92d4257b30f01b62d4ef941686b7e0138a8c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fe3e92d4257b30f01b62d4ef941686b7e0138a8c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813156" author="xgen-internal-githook" created="Mon, 26 Jan 2015 16:38:36 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;erh&apos;, u&apos;name&apos;: u&apos;Eliot Horowitz&apos;, u&apos;email&apos;: u&apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: better tcmalloc cleanup&lt;/p&gt;

&lt;p&gt;(cherry picked from commit c4ec7db6ce25e7147ae5f46ac0a7f6a52a0b4c3e)&lt;br/&gt;
Branch: v3.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3ef65d17fd8081cb65330bf90bffd093e7fd020b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3ef65d17fd8081cb65330bf90bffd093e7fd020b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="813148" author="xgen-internal-githook" created="Mon, 26 Jan 2015 16:34:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;erh&apos;, u&apos;name&apos;: u&apos;Eliot Horowitz&apos;, u&apos;email&apos;: u&apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: better tcmalloc cleanup&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c4ec7db6ce25e7147ae5f46ac0a7f6a52a0b4c3e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c4ec7db6ce25e7147ae5f46ac0a7f6a52a0b4c3e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="810746" author="xgen-internal-githook" created="Thu, 22 Jan 2015 17:24:43 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;GeertBosch&apos;, u&apos;name&apos;: u&apos;Geert Bosch&apos;, u&apos;email&apos;: u&apos;geert@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16773&quot; title=&quot;Performance degradation due to TCMalloc scalability&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16773&quot;&gt;&lt;del&gt;SERVER-16773&lt;/del&gt;&lt;/a&gt;: Mark threads idle for tcmalloc&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4c83a604004c329ac114c53df38ac96421ebcf83&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4c83a604004c329ac114c53df38ac96421ebcf83&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="807767" author="john.page" created="Mon, 19 Jan 2015 18:15:24 +0000"  >&lt;p&gt;Does rc4 and rc5 show a difference?&lt;/p&gt;

&lt;p&gt;&lt;b&gt;{&lt;/b&gt; name     : &quot;John Page&quot;,&lt;br/&gt;
  title    : &quot;Senior Solutions Architect&quot;,&lt;br/&gt;
  phone    : &quot;(44) 755 737 1664&quot;,&lt;br/&gt;
  location : &quot;Glasgow, UK&quot;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;}&lt;/b&gt;&lt;/p&gt;



&lt;p&gt;On Mon, Jan 19, 2015 at 6:13 PM, Geert Bosch (JIRA) &amp;lt;jira@mongodb.org&amp;gt;&lt;/p&gt;
</comment>
                            <comment id="807764" author="john.page" created="Mon, 19 Jan 2015 18:09:16 +0000"  >&lt;p&gt;If you want me to retest I&apos;m happy to do so. 2.8-rc5?&lt;/p&gt;</comment>
                            <comment id="807761" author="geert.bosch" created="Mon, 19 Jan 2015 18:07:06 +0000"  >&lt;p&gt;I just redid a run with a build of vanilla rc5 without any patches and when doing a 1500 second run didn&apos;t see any performance anomalies, will attach results graph.&lt;/p&gt;

&lt;p&gt;server build info:&lt;br/&gt;
&amp;gt; db.serverBuildInfo()&lt;br/&gt;
{&lt;br/&gt;
        &quot;version&quot; : &quot;2.8.0-rc5&quot;,&lt;br/&gt;
        &quot;gitVersion&quot; : &quot;74b351de21c84438b12a83b28e155f5e69e3c1eb&quot;,&lt;br/&gt;
        &quot;OpenSSLVersion&quot; : &quot;&quot;,&lt;br/&gt;
        &quot;sysInfo&quot; : &quot;Linux gouda 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49&quot;,&lt;br/&gt;
        &quot;loaderFlags&quot; : &quot;-fPIC -pthread -Wl,-z,now -rdynamic&quot;,&lt;br/&gt;
        &quot;compilerFlags&quot; : &quot;-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Werror -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fno-builtin-memcmp -std=c99&quot;,&lt;br/&gt;
        &quot;allocator&quot; : &quot;tcmalloc&quot;,&lt;br/&gt;
        &quot;versionArray&quot; : [&lt;br/&gt;
                2,&lt;br/&gt;
                8,&lt;br/&gt;
                0,&lt;br/&gt;
                -5&lt;br/&gt;
        ],&lt;br/&gt;
        &quot;javascriptEngine&quot; : &quot;V8&quot;,&lt;br/&gt;
        &quot;bits&quot; : 64,&lt;br/&gt;
        &quot;debug&quot; : false,&lt;br/&gt;
        &quot;maxBsonObjectSize&quot; : 16777216,&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;br/&gt;
&amp;gt; &lt;/p&gt;</comment>
                            <comment id="807674" author="john.page" created="Mon, 19 Jan 2015 16:16:25 +0000"  >&lt;p&gt;I don&apos;t have any answers for that one.&lt;/p&gt;</comment>
                            <comment id="807672" author="geert.bosch" created="Mon, 19 Jan 2015 16:13:12 +0000"  >&lt;p&gt;OK, thanks. I&apos;ll resolve that, however this does not seem to invalidate the results so far. Do you have any idea why the first 450 seconds of a 600 second run would look different from the first 450 seconds of a 1500 second run?&lt;/p&gt;</comment>
                            <comment id="807181" author="john.page" created="Sat, 17 Jan 2015 19:47:24 +0000"  >&lt;p&gt;What are you seeing is where you don&apos;t have enough client side file handles&lt;br/&gt;
in unlimited.  Processes in the loader die after n seconds unless there is&lt;br/&gt;
an error from the server.&lt;/p&gt;
</comment>
                            <comment id="807147" author="john.page" created="Sat, 17 Jan 2015 17:09:24 +0000"  >&lt;p&gt;That&apos;s the C driver code for not enough available file handles.&lt;/p&gt;
</comment>
                            <comment id="807134" author="geert.bosch" created="Sat, 17 Jan 2015 15:14:16 +0000"  >&lt;p&gt;There seems a problem with the load generator: the first 500 seconds look much better if I specify a test duration of 1500 seconds. There also an issue with not all threads completing always: if I specify 1000 threads the test might finish with 250 or so. Error message is: &quot;Failed to read 4 bytes from socket. Child Quitl.&quot; The test system has plenty of resources to run and is never overloaded.&lt;/p&gt;

&lt;p&gt;Anyway, I get good throughput now with regular journaling/yielding etc. enabled for a 1500 second (25 min) test run.&lt;/p&gt;</comment>
                            <comment id="806668" author="david.daly" created="Fri, 16 Jan 2015 18:34:06 +0000"  >&lt;p&gt;I tried John&apos;s workload against RC4. Interesting behavior. &lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/61957/61957_rc4.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;After about 3-4 minutes, the performance increases. I&apos;ve labelled that as time B. Test starts at A. Seems stable after B. &lt;/p&gt;</comment>
                            <comment id="806487" author="david.daly" created="Fri, 16 Jan 2015 16:24:13 +0000"  >&lt;p&gt;Short summary:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Query only workload does not show degradation&lt;/li&gt;
	&lt;li&gt;Update only workload shows a slow degradation over time&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="806415" author="john.page" created="Fri, 16 Jan 2015 15:28:00 +0000"  >&lt;p&gt;I will depend what version of that code you have.&lt;br/&gt;
Older code opened another X connections to read the test to see if it had changed then dropped them again - newer ones use the same connection throughout so with 1000 threads its a constant 1001 connections not 1001-&amp;gt;2001-&amp;gt;1001&lt;/p&gt;</comment>
                            <comment id="806406" author="david.daly" created="Fri, 16 Jan 2015 15:23:14 +0000"  >&lt;p&gt;I think the spike in inserts is an artifact of the restart and the workload reconnection. I think the workload fails to get the test specification from the database, and then runs the default workload for a little while, until it suceeds in querying the server for the current test configuration. &lt;/p&gt;</comment>
                            <comment id="806378" author="david.daly" created="Fri, 16 Jan 2015 15:02:22 +0000"  >&lt;p&gt;Looking at the stats, I see the connections go to zero when I restart it, but then it goes back to where it was after I resume the workload. &lt;/p&gt;</comment>
                            <comment id="806375" author="david.daly" created="Fri, 16 Jan 2015 14:57:36 +0000"  >&lt;p&gt;In case it&apos;s interesting, here&apos;s a shot of the spike in inserts after restart&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/61924/61924_WorkloadSuspension.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;A: Start workload with 512 threads&lt;/li&gt;
	&lt;li&gt;B: Turn off inserts&lt;/li&gt;
	&lt;li&gt;C: Suspend workload&lt;/li&gt;
	&lt;li&gt;D: Resume workload&lt;/li&gt;
	&lt;li&gt;E: Suspend workload and restart server&lt;/li&gt;
	&lt;li&gt;F: Resume workload&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also, opcounters for query and update seem higher than documents returned and documents updated. &lt;/p&gt;</comment>
                            <comment id="806371" author="john.page" created="Fri, 16 Jan 2015 14:53:24 +0000"  >&lt;p&gt;I don&apos;t think the workload will reconnect if you restart the server though&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;it build a connection set up front.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;{&lt;/b&gt; name     : &quot;John Page&quot;,&lt;br/&gt;
  title    : &quot;Senior Solutions Architect&quot;,&lt;br/&gt;
  phone    : &quot;(44) 755 737 1664&quot;,&lt;br/&gt;
  location : &quot;Glasgow, UK&quot;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;}&lt;/b&gt;&lt;/p&gt;



</comment>
                            <comment id="806368" author="david.daly" created="Fri, 16 Jan 2015 14:49:54 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=john.page&quot; class=&quot;user-hover&quot; rel=&quot;john.page&quot;&gt;john.page&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;Trying a slightly different tack now. Doing a long duration run, and the suspending the workload process to restart the server. &lt;br/&gt;
One interesting thing (maybe artifact) is I see a burst of inserts after unsuspending the workload. Going to try various loads and see what I can find. &lt;/p&gt;</comment>
                            <comment id="806198" author="john.page" created="Fri, 16 Jan 2015 06:37:49 +0000"  >&lt;p&gt;Query and update only access records inserted in that run so a query update only workload isn&apos;t doing much. &lt;/p&gt;</comment>
                            <comment id="806096" author="david.daly" created="Fri, 16 Jan 2015 02:24:08 +0000"  >&lt;p&gt;The workload can be adjusted between inserts, queries, and updates by adjusting the entry in the testsrv.test collection on the system under test. There is one record with _id : &quot;loadtest&quot;. The number of operations of each type is proportional to the value in each field. Set &quot;insert&quot; : 0 to stop all inserts. Fields can be updated while the test is running. &lt;/p&gt;

&lt;p&gt;Some initial observations from adjusting the running load (all with 512 threads)&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Ran the default workload and saw the performance drop off.&lt;/li&gt;
	&lt;li&gt;Set inserts to zero: saw the inserts stop, but the performance of the queries and updates did not appreciably improve.&lt;/li&gt;
	&lt;li&gt;After idling the system, the mixed query/update workload without inserts showed significantly higher performance.&lt;/li&gt;
	&lt;li&gt;The query/update workload performance is not dropping off after the idle period. 10 minute run&lt;/li&gt;
	&lt;li&gt;Restarted server and ran quey/update workload. It showed stable performance.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="805833" author="geert.bosch" created="Thu, 15 Jan 2015 21:48:22 +0000"  >&lt;p&gt;Interestingly, jemalloc  doesn&apos;t get as bad but shows same shape. LockLessInc malloc is similar as well.&lt;/p&gt;</comment>
                            <comment id="805633" author="geert.bosch" created="Thu, 15 Jan 2015 18:57:20 +0000"  >&lt;p&gt;TCMalloc 2.0 has similar behavior as well.&lt;/p&gt;</comment>
                            <comment id="805441" author="geert.bosch" created="Thu, 15 Jan 2015 16:20:46 +0000"  >&lt;p&gt;TCMalloc 2.4 has same behavior as the 2.2 version we are using now. Will be trying 2.0 next.&lt;/p&gt;</comment>
                            <comment id="801705" author="dan@10gen.com" created="Sat, 10 Jan 2015 00:52:45 +0000"  >&lt;p&gt;John, can you include the git hash of the mongod binaries you&apos;re testing as well as the parameters you&apos;re passing to mongod?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="179037">SERVER-16879</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="169126">SERVER-16131</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="226893">SERVER-20104</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="179037">SERVER-16879</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="266795">SERVER-22763</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="455005">SERVER-31839</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="62296" name="MMAP-26.csv.ops.png" size="207035" author="john.page@mongodb.com" created="Tue, 20 Jan 2015 23:19:09 +0000"/>
                            <attachment id="62297" name="MMAP-28.csv.ops.png" size="261511" author="john.page@mongodb.com" created="Tue, 20 Jan 2015 23:19:09 +0000"/>
                            <attachment id="61924" name="WorkloadSuspension.png" size="49580" author="david.daly@mongodb.com" created="Fri, 16 Jan 2015 14:57:36 +0000"/>
                            <attachment id="61957" name="rc4.png" size="71286" author="david.daly@mongodb.com" created="Fri, 16 Jan 2015 18:34:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>34.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 8 Jan 2015 15:01:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 7 weeks, 4 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 7 weeks, 4 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>david.daly@mongodb.com</customfieldvalue>
            <customfieldvalue>wolf_kdy</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>igor</customfieldvalue>
            <customfieldvalue>john.page@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlfbj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs59uf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>155782</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs6bkf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>