<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:48:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18788] wiredtiger 10x slower than mmap replica instance </title>
                <link>https://jira.mongodb.org/browse/SERVER-18788</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When we switch primaries to be the wiredTiger instances, while there&lt;br/&gt;
is almost no additional stress on the machines, the time taken for&lt;br/&gt;
queries goes out of whack.&lt;/p&gt;

&lt;p&gt;For our homepage, &lt;a href=&quot;https://www.shopcade.com/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.shopcade.com/&lt;/a&gt;, to load for a logged in&lt;br/&gt;
user (me), about 200 mongo queries execute in 200 ms. Please find&lt;br/&gt;
attached a screen capture (homepage-mongo-debug.png) from our in-house&lt;br/&gt;
web-request level profiling tool.&lt;br/&gt;
When we switch our primaries to the wiredTiger instances, the time on&lt;br/&gt;
average for each query goes up by a factor of 10/20 times. There&lt;br/&gt;
aren&apos;t a few queries that are slow, but all of them, compared to MMAP.&lt;br/&gt;
The total load time for the page goes from 1-2 seconds into 10s of&lt;br/&gt;
seconds.&lt;/p&gt;

&lt;p&gt;I&apos;ve got only marginal improvements changing to the following configuration&lt;br/&gt;
wiredTiger:&lt;br/&gt;
    collectionConfig:&lt;br/&gt;
       blockCompressor: &quot;none&quot;&lt;br/&gt;
    engineConfig:&lt;br/&gt;
       cacheSizeGB: 180&lt;br/&gt;
       journalCompressor: &quot;none&quot;&lt;br/&gt;
    indexConfig:&lt;br/&gt;
       prefixCompression: false&lt;/p&gt;</description>
                <environment>* mongodb 3.0.3&lt;br/&gt;
* 4x 244GiB servers on Amazon&amp;#39;s cloud (r3.8xlarge running Ubuntu 14.04.1 LTS) &lt;br/&gt;
* 2 shards, s01 and s02&lt;br/&gt;
db.shards.find()&lt;br/&gt;
{ &amp;quot;_id&amp;quot; : &amp;quot;s01&amp;quot;, &amp;quot;host&amp;quot; :&lt;br/&gt;
&amp;quot;s01/s01r01.mongo.shopcade.com:27018,s01r02.mongo.shopcade.com:27018&amp;quot;,&lt;br/&gt;
&amp;quot;tags&amp;quot; : [ &amp;quot;main&amp;quot; ] }&lt;br/&gt;
{ &amp;quot;_id&amp;quot; : &amp;quot;s02&amp;quot;, &amp;quot;host&amp;quot; :&lt;br/&gt;
&amp;quot;s02/s02r01.mongo.shopcade.com:27018,s02r02.mongo.shopcade.com:27018&amp;quot;,&lt;br/&gt;
&amp;quot;tags&amp;quot; : [ &amp;quot;products&amp;quot; ] }&lt;br/&gt;
* s01r01 and s02r01 are MMAP&lt;br/&gt;
* s01r02 and s02r02 are wiredTiger.&lt;br/&gt;
The MMAP instances are primaries.&lt;br/&gt;
</environment>
        <key id="207827">SERVER-18788</key>
            <summary>wiredtiger 10x slower than mmap replica instance </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="sam.kleinman">Sam Kleinman</assignee>
                                    <reporter username="vain">navin kumar</reporter>
                        <labels>
                    </labels>
                <created>Tue, 2 Jun 2015 09:13:31 +0000</created>
                <updated>Mon, 15 Jun 2015 15:44:38 +0000</updated>
                            <resolved>Mon, 15 Jun 2015 15:44:38 +0000</resolved>
                                                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="940558" author="samk" created="Mon, 15 Jun 2015 15:44:19 +0000"  >&lt;p&gt;Glad to hear that you resolved this issue!&lt;/p&gt;

&lt;p&gt;Cheers,&lt;br/&gt;
sam&lt;/p&gt;</comment>
                            <comment id="940519" author="vain" created="Mon, 15 Jun 2015 15:25:15 +0000"  >&lt;p&gt;Problem solved!&lt;br/&gt;
I have now tested for a few days. All queries now have acceptable times. Thank you very much for your help, Sam, Asya, Keith.&lt;/p&gt;

&lt;p&gt;Notes:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The single biggest problem was the large number of immortal cursors&lt;/li&gt;
	&lt;li&gt;enabled block compression and prefix compression, with no drop in performance&lt;/li&gt;
	&lt;li&gt;eliminated/optimised most scanning queries - they were mostly from our backend cron jobs, (oddly, MMAP seemed to have acceptable times on large scans)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Best,&lt;br/&gt;
Navin&lt;/p&gt;</comment>
                            <comment id="934925" author="vain" created="Tue, 9 Jun 2015 10:47:59 +0000"  >&lt;p&gt;Sam, Asya,&lt;br/&gt;
Thank you very much for your feedback.&lt;/p&gt;

&lt;p&gt;I&apos;m working on your inputs, and have varying results at this point. Will get back to you with performance data. Fingers crossed!&lt;/p&gt;</comment>
                            <comment id="931937" author="samk" created="Thu, 4 Jun 2015 18:56:12 +0000"  >&lt;p&gt;Reviewing your logs, it looks like the MMAP is subject to similar kinds of slow queries: inefficient use of an index or collection scans, etc. as on WiredTiger. This is expected, and while this isn&apos;t the issue you&apos;re experiencing, I&apos;m confident that you&apos;ll see improved and more consistent performance if you modify your indexes or queries. This is outside of the scope of this ticket, but the following documentation may help: &lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;a href=&quot;http://docs.mongodb.org/manual/applications/indexes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Indexing Strategies&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;http://docs.mongodb.org/manual/tutorial/analyze-query-plan/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Analyze Query Performance&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Second, the data size you provided looks like is from the output of db.stats(), I just want to confirm that you&apos;ve included a sum of all databases on this replica set. The main difference that I can see between these deployments is that the MMAP instance effectively has the whole machine&apos;s ram to use for in-memory cache, if the entire data set fits in RAM, as it looks like it does from these numbers. At the same time the WiredTiger instance is limited to 180 GB, and so there&apos;s more churn on the cache, which could explain the performance effects you&apos;re seeing. You could restart the wiredTiger instance with a larger cache (i.e. 230-240 GB) and see if that resolves the issue. &lt;/p&gt;

&lt;p&gt;If it turns out that this is a cache size issue; in the future, with better index utilization you may be able to reduce the cache size alloted to MongoDB. &lt;/p&gt;

&lt;p&gt;Third, it looks like your instance has a large number of open cursors, which can impact performance. From the &lt;tt&gt;serverStatus&lt;/tt&gt; output that you include, it looks like your application opened about 10,000 curors with the &lt;tt&gt;noTimeout&lt;/tt&gt; option at the top of the hour, and about 10,000 other cursors are timing out every hour. If you can modify your application to &lt;b&gt;not&lt;/b&gt; open cursors with the &lt;tt&gt;noTimeout&lt;/tt&gt; option, you may see improved performance. Shortening timeouts or closing cursors when the application is done with them may also help.&lt;/p&gt;

&lt;p&gt;As you test this, if you end up with a wiredTiger primary and MMAP secondaries, I might encourage you to increase the write concern: wiredTiger can typically handle a higher volume of writes than MMAP, and you should make sure that the secondaries can replicate operations fast enough. &lt;/p&gt;

&lt;p&gt;In general, we would expect a database server running wiredTiger to use more CPU resources (particularly with compression) than one running MMAP: as a result the differences in system load reporting may not indicate an issue.&lt;/p&gt;

&lt;p&gt;To summarize, I think we may be able to explain the performance effect you&apos;re seeing, may be attributable to the following factors:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;cache size,&lt;/li&gt;
	&lt;li&gt;inefficient queries and table scans.&lt;/li&gt;
	&lt;li&gt;number of open cursors.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I hope this helps. At this point I&apos;m not inclined to think that there is a specific bug with MongoDB+WT. Please let us know if you still see a performance degradation with wiredTiger after modifying your application and deployment.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
sam&lt;/p&gt;</comment>
                            <comment id="931576" author="asya" created="Thu, 4 Jun 2015 14:55:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vain&quot; class=&quot;user-hover&quot; rel=&quot;vain&quot;&gt;vain&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I happened to notice that you seem to have a large number of cursors that don&apos;t get closed - there seem to be quite a few that time out after being idle for the timeout period, and it looks like you have a large number of &quot;noTimeout&quot; cursors accumulating after a while.   Can you explain how you are using cursors in the application code?&lt;/p&gt;

&lt;p&gt;Asya&lt;/p&gt;</comment>
                            <comment id="930344" author="vain" created="Wed, 3 Jun 2015 13:09:15 +0000"  >&lt;p&gt;Sam, &lt;br/&gt;
Thanks for your reply.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;1. Do these queries and count operations also have similar characteristics with an MMAP primary? Could you upload a similar segment of the log from a primary that uses MMAP?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;no, on MMAP, there are hardly any slow queries. more details below&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;2. Would it be possible to paste the output of db.getIndexes(); in this ticket?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I&apos;ve added it in the attached zip file&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;3. Did you observe this effect when compression was enabled? Why inspired you to change this setting?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;No, turning off compression was something I was testing - not sure it had any impact&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;4. For your write operations, what level of write concern do you specify?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;1 - Acknowledged&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;5. How much RAM does the system have? How much data is stored in MongoDB?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;each of our mongo servers have 244GiB of RAM. For shard s01:&lt;br/&gt;
on the wiredTiger instance:&lt;br/&gt;
&amp;gt;	&quot;dataSize&quot; : 100168049112,&lt;br/&gt;
&amp;gt;	&quot;storageSize&quot; : 106178719744,&lt;br/&gt;
&amp;gt;	&quot;indexSize&quot; : 48765927424,&lt;br/&gt;
on the MMAP instance:&lt;br/&gt;
&amp;gt;	&quot;dataSize&quot; : 163739003696,&lt;br/&gt;
&amp;gt;	&quot;storageSize&quot; : 230548450400,&lt;br/&gt;
&amp;gt;	&quot;indexSize&quot; : 87954497344,&lt;/p&gt;

&lt;p&gt;This morning, to generate more data, I switched the primary again to be wiredTiger. It took about half an hour for performance to drop for the same queries. I&apos;ve attached details from this time below. Do have a look inside the &lt;b&gt;mongo-query-times-compared&lt;/b&gt; directory. I am sorry to upload such large images, but I feel they best illustrate the problem. &lt;/p&gt;

&lt;p&gt;Hope this helps, and thank you very much.&lt;/p&gt;

&lt;p&gt;-----------------------&lt;br/&gt;
&lt;b&gt;files in mongo-engine-comparisons.zip&lt;/b&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;db-indexes.txt&lt;/li&gt;
		&lt;li&gt;db-serverStatus-polled-every-minute&lt;/li&gt;
		&lt;li&gt;mmap.log&lt;/li&gt;
		&lt;li&gt;wiredTiger.log&lt;/li&gt;
		&lt;li&gt;system-load-average-both-machines.png&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;the following are query times for a page load:&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;mongo-query-times-compared/0911-UTC-homepage-mmap.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/0922-UTC-homepage-wiredTiger-on-switchover.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/0934-UTC-homepage-wiredTiger-2.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/1006-UTC-homepage-wiredTiger-8.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/1008-UTC-homepage-wiredTiger-9.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/1012-UTC-homepage-back-to-mmap.png&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;more like the above, can ignore&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;mongo-query-times-compared/more-page-query-times/0943-UTC-homepage-wiredTiger-3.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/more-page-query-times/0951-UTC-homepage-wiredTiger-4.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/more-page-query-times/0955-UTC-homepage-wiredTiger-5.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/more-page-query-times/0959-UTC-homepage-wiredTiger-6.png&lt;/li&gt;
		&lt;li&gt;mongo-query-times-compared/more-page-query-times/1003-UTC-homepage-wiredTiger-7.png&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="929572" author="vain" created="Tue, 2 Jun 2015 16:13:37 +0000"  >&lt;p&gt;After turning off prefix compression, re-syncing data from scratch, and making wiredTiger primary, there was only about an hour of fast performance. That was at the time of my tweet. However query times quickly deteriorated after that. So I doubt prefix compression is the issue. I jumped the gun with that tweet, apologies.&lt;/p&gt;</comment>
                            <comment id="929530" author="keith.bostic" created="Tue, 2 Jun 2015 15:31:53 +0000"  >&lt;p&gt;Navin, your original tweet made me think prefix compression was an issue, was that based on measurements with/without index prefix compression configured? Or did that not have much of an effect?&lt;/p&gt;</comment>
                            <comment id="929524" author="samk" created="Tue, 2 Jun 2015 15:26:28 +0000"  >&lt;p&gt;From looking over the logs I see a number of factors that might contribute to degraded performance. I&apos;m not sure if these can, alone, explain the performance impact that you see, but it would be good to remove potential factors as we troubleshoot. &lt;/p&gt;

&lt;p&gt;First,  I see some queries that are not able to use an index effectively, (e.g. high absolute &lt;tt&gt;nscannedObjects&lt;/tt&gt; values, high ratios of &lt;tt&gt;nscannedObjects&lt;/tt&gt; to &lt;tt&gt;nscanned&lt;/tt&gt;, and &lt;tt&gt;scanAndOrder&lt;/tt&gt; operations which do in-memory sorts.) For example: &lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-05-29T15:58:25.689+0000 I QUERY    [conn17848] query shopcade.lists query: { $query: { pc: { $gte: 6 } }, $orderby: { lpa: -1 } } planSummary: IXSCAN { pc: 1.0, lpa: -1.0 }, IXSCAN { pc: 1.0, lpa: -1.0 } cursorid:322462517267 ntoreturn:5 ntoskip:0 nscanned:26912 nscannedObjects:26912 scanAndOrder:1 keyUpdates:0 writeConflicts:0 numYields:210 nreturned:5 reslen:895 locks:{ Global: { acquireCount: { r: 211 } }, Database: { acquireCount: { r: 211 } }, Collection: { acquireCount: { r: 211 } } } 339ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-05-29T15:59:40.650+0000 I QUERY    [conn17935] query shopcade.brands query: { $query: { nb_prods.GB: { $gte: 3 }, _id: { $regex: &quot;^C&quot;, $options: &quot;i&quot; } }, $orderby: { pop: -1 } } planSummary: IXSCAN { nb_prods.GB: 1.0, pop: -1.0 } cursorid:133281854156 ntoreturn:0 ntoskip:0 nscanned:21900 nscannedObjects:21900 scanAndOrder:1 keyUpdates:0 writeConflicts:0 numYields:171 nreturned:101 reslen:11939 locks:{ Global: { acquireCount: { r: 172 } }, Database: { acquireCount: { r: 172 } }, Collection: { acquireCount: { r: 172 } } } 109ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-05-29T16:08:46.118+0000 I QUERY    [conn17911] query shopcade.brands query: { $query: { nb_prods.US: { $gte: 3 }, _id: { $regex: &quot;^O&quot;, $options: &quot;i&quot; } }, $orderby: { pop: -1 } } planSummary: IXSCAN { nb_prods.US: 1.0, pop: -1.0 } cursorid:135280524753 ntoreturn:0 ntoskip:0 nscanned:34148 nscannedObjects:34148 scanAndOrder:1 keyUpdates:0 writeConflicts:0 numYields:267 nreturned:101 reslen:9683 locks:{ Global: { acquireCount: { r: 268 } }, Database: { acquireCount: { r: 268 } }, Collection: { acquireCount: { r: 268 } } } 138ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Second there a large number of count operations that must inspect a many number of documents, and are hitting yields:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-05-29T16:00:39.492+0000 I COMMAND  [conn17838] command shopcade.$cmd command: count { count: &quot;followers&quot;, query: { following: ObjectId(&apos;524497ff458719e82cd7d5e0&apos;), type: &quot;user&quot; } } planSummary: IXSCAN { following: 1.0, _id: -1.0 } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:491 reslen:104 locks:{ Global: { acquireCount: { r: 492 } }, Database: { acquireCount: { r: 492 } }, Collection: { acquireCount: { r: 492 } } } 121ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-05-29T17:15:17.175+0000 I COMMAND  [conn17753] command shopcade.$cmd command: count { count: &quot;activity_collector&quot;, query: { _id: { $gte: ObjectId(&apos;5565e33b0000000000000000&apos;) }, a: ObjectId(&apos;51a4dde8458719bc3a0002ad&apos;), ac: { $in: [ 60 ] } } } planSummary: IXSCAN { a: 1.0, _id: -1.0 } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:104 locks:{ Global: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 206419 } }, Collection: { acquireCount: { r: 1 } } } 206ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;ol&gt;
	&lt;li&gt;Do these queries and count operations also have similar characteristics with an MMAP primary? Could you upload a similar segment of the log from a primary that uses MMAP?&lt;/li&gt;
	&lt;li&gt;Would it be possible to paste the output of &lt;tt&gt;db.getIndexes();&lt;/tt&gt; in this ticket?&lt;/li&gt;
	&lt;li&gt;Did you observe this effect when compression was enabled? Why inspired you to change this setting?&lt;/li&gt;
	&lt;li&gt;For your write operations, what level of write concern do you specify?&lt;/li&gt;
	&lt;li&gt;How much RAM does the system have? How much data is stored in MongoDB?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thanks for your patience, and I hope we can get to the bottom of this quickly.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;br/&gt;
sam&lt;/p&gt;</comment>
                            <comment id="929447" author="vain" created="Tue, 2 Jun 2015 13:35:41 +0000"  >&lt;p&gt;before switching to the configuration, i had all values at their defaults.&lt;/p&gt;

&lt;p&gt;I&apos;ve attached the mongo log for when wiredTiger was primary&lt;/p&gt;</comment>
                            <comment id="929426" author="ramon.fernandez" created="Tue, 2 Jun 2015 12:51:03 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vain&quot; class=&quot;user-hover&quot; rel=&quot;vain&quot;&gt;vain&lt;/a&gt;, since this issue pertains to the MongoDB server we&apos;ve moved the ticket to the SERVER project.&lt;/p&gt;

&lt;p&gt;Do you have logs from the WiredTiger primary when serving those slow queries? What configuration where you using before switching to the one above?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="185927">SERVER-17364</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="76719" name="Screenshot 2015-06-04 10.33.37.png" size="82416" author="asya.kamsky@mongodb.com" created="Thu, 4 Jun 2015 14:47:56 +0000"/>
                            <attachment id="76720" name="Screenshot 2015-06-04 10.38.20.png" size="86716" author="asya.kamsky@mongodb.com" created="Thu, 4 Jun 2015 14:47:56 +0000"/>
                            <attachment id="76586" name="mongo-engine-comparisons.zip" size="51691784" author="vain" created="Wed, 3 Jun 2015 13:09:15 +0000"/>
                            <attachment id="76480" name="mongod.log" size="511145" author="vain" created="Tue, 2 Jun 2015 13:34:52 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 2 Jun 2015 12:51:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 35 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>sam.kleinman</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 35 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>keith.bostic@mongodb.com</customfieldvalue>
            <customfieldvalue>vain</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>sam.kleinman</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl49z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs9w3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs5ozj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>