<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:03:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-23433] High acquire time for Global Intent Shared lock using WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-23433</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I&apos;m seeing a number of queries that are experiencing very high execution times, almost entirely due to timeAcquiringMicros against the Global Intent Shared lock.&lt;/p&gt;

&lt;p&gt;Some examples of the entries from the logs (server, db and collection names anonymised):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Mar 30 15:17:33 SERVER mongod.27017[1461]: [conn2254368] command DB.COLLECTION command: count { count: &quot;requestsByUserState&quot;, query: { timeGroup: &quot;2016-03-24&quot; } } planSummary: COUNT_SCAN { timeGroup: -1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:62 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 32914737 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 32915ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Mar 30 15:25:06 SERVER mongod.27017[1461]: [conn1899344] command DB.COLLECTION command: aggregate { aggregate: &quot;scriptLogLevelsCount.DAILY&quot;, pipeline: [ { $match: { _id: { $gte: &quot;2016-03-30&quot;, $lt: &quot;2016-03-31&quot; } } }, { $group: { _id: null, FATAL: { $sum: &quot;$scriptLogLevelsCount.FATAL&quot; }, ERROR: { $sum: &quot;$scriptLogLevelsCount.ERROR&quot; }, WARN: { $sum: &quot;$scriptLogLevelsCount.WARN&quot; }, INFO: { $sum: &quot;$scriptLogLevelsCount.INFO&quot; }, DEBUG: { $sum: &quot;$scriptLogLevelsCount.DEBUG&quot; } } } ] } keyUpdates:0 writeConflicts:0 numYields:1 reslen:138 locks:{ Global: { acquireCount: { r: 8 }, acquireWaitCount: { r: 4 }, timeAcquiringMicros: { r: 125415124 } }, Database: { acquireCount: { r: 4 } }, Collection: { acquireCount: { r: 4 } } } protocol:op_query 125416ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Mar 30 16:03:07 SERVER mongod.27017[1461]: [conn2261655] command DB.COLLECTION command: getMore { getMore: 266905304107253, collection: &quot;COLLECTION&quot; } cursorid:266905304107253 keyUpdates:0 writeConflicts:0 numYields:36 nreturned:4724 reslen:4195106 locks:{ Global: { acquireCount: { r: 74 }, acquireWaitCount: { r: 8 }, timeAcquiringMicros: { r: 88559517 } }, Database: { acquireCount: { r: 37 } }, Collection: { acquireCount: { r: 37 } } } protocol:op_query 88570ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We&apos;re seeing thousands of these per day, from what I can see always on secondary servers, but in this environment we perform the majority of our queries against the secondaries. (Its an analytics processing platform, so is write-heavy on the master and it doesn&apos;t matter if queries are slightly stale)&lt;/p&gt;

&lt;p&gt;When the slow queries happen, there are often a number of them clustered together during the same or subsequent seconds, which makes me wonder if this is a WiredTiger checkpoint? There are a lot of databases in this replica set, meaning a significant number of files to checkpoint.&lt;/p&gt;

&lt;p&gt;Environment details:&lt;br/&gt;
3 node replica set&lt;br/&gt;
MongoDB 3.2.3 64-bit&lt;br/&gt;
WiredTiger storage engine (default config apart from &quot;directoryForIndexes: true&quot;)&lt;br/&gt;
Ubuntu 14.04.3 LTS&lt;br/&gt;
Azure VMs - Standard DS13 (8 Cores, 56 GB memory)&lt;br/&gt;
~5,700 databases&lt;br/&gt;
~293,000 WiredTiger data and index files&lt;/p&gt;</description>
                <environment></environment>
        <key id="276069">SERVER-23433</key>
            <summary>High acquire time for Global Intent Shared lock using WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="gregmurphy">Greg Murphy</reporter>
                        <labels>
                    </labels>
                <created>Wed, 30 Mar 2016 16:22:52 +0000</created>
                <updated>Thu, 24 Aug 2017 14:21:12 +0000</updated>
                            <resolved>Tue, 17 May 2016 19:17:34 +0000</resolved>
                                    <version>3.2.3</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="1266752" author="thomas.schubert" created="Tue, 17 May 2016 19:17:34 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed workload description. The issue here is that the checkpoint handle aquire phase will block opening a cursor on any table if it is the first time that table is being opened in a particular WiredTiger session. When there are a smaller number of collections/indexes, MongoDB caches cursors to all tables, and thus operations generally do not need to open tables. However, the size of those caches is limited (to keep them efficient, and avoid memory overhead) which leads to apply ops needing to open tables. We are working on &lt;a href=&quot;https://jira.mongodb.org/browse/WT-1598&quot; title=&quot;Remove the schema, table locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-1598&quot;&gt;&lt;del&gt;WT-1598&lt;/del&gt;&lt;/a&gt; to improve this behavior.&lt;/p&gt;

&lt;p&gt;Please feel to vote for &lt;a href=&quot;https://jira.mongodb.org/browse/WT-1598&quot; title=&quot;Remove the schema, table locks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-1598&quot;&gt;&lt;del&gt;WT-1598&lt;/del&gt;&lt;/a&gt; and watch it for updates.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1238819" author="gregmurphy" created="Mon, 18 Apr 2016 16:09:40 +0000"  >&lt;p&gt;Sure - its easier to describe narratively rather than answering the questions in order, but hopefully this covers everything:&lt;/p&gt;

&lt;p&gt;This particular MongoDB cluster contains analytics data for our multi-tenant platform, with a database per customer (4,728 of those at the moment). Depending on the data generated by that customer, each database can contain up to 55 collections and 58 indexes. The data ingested into the database is generated every 10 minutes and ingested as close to generation as possible. All customers don&apos;t continually generate data, so some databases are more active than others.&lt;/p&gt;

&lt;p&gt;During the ingestion process (which is parallelised with 7 processes running concurrently, each one working on a subset of databases) the current data state is read into memory by the process from the secondary MongoDB nodes. The ingestion process then merges this with the incoming data and persists the results back to MongoDB before performing some aggregations and persisting the results of them. This persistence results in updates/inserts happening on the against the majority of the collections in the database being processed, but as these are aggregated results the number of documents affected is low.&lt;/p&gt;

&lt;p&gt;New databases, collections and indexes are only created when new customers arrive. In the last 8 hours there have been 58 new databases created, so not a high frequency of object creations.&lt;/p&gt;

&lt;p&gt;Its difficult to say precisely how many collections and indexes are actively in use at any time as it depends on the processing point that each ingestion process is at. Also end-user queries can hit this cluster (from a web portal).&lt;/p&gt;

&lt;p&gt;Hope that helps, please let me know if I&apos;ve missed anything.&lt;/p&gt;</comment>
                            <comment id="1238662" author="thomas.schubert" created="Mon, 18 Apr 2016 14:49:03 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The number of connection data handles that are currently active (around 200 thousand) is very large is likely explains the issue you are experiencing. When WiredTiger creates a checkpoint it needs to look at each open table (collection or index). To ensure an unchanging set of tables, it takes a lock during this process. Consequently, this process may block other operations including a checkpoint. It is worth noting that the code that gathers these handles was modified in the 3.2.5 release, which explains the difference in the behavior you observed.&lt;/p&gt;

&lt;p&gt;Please see the attached FTDC graph from your node on MongoDB 3.2.5. We can see that the spikes in read time acquiring and FTDC dropouts are aligned with when checkpoints start.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/118086/118086_s23433.png&quot; width=&quot;700&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;To continue to investigate, can you please describe some details about your workload? In particular please be sure to answer the following questions:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;How many collections and indexes are actively in use at a particular point in time?&lt;/li&gt;
	&lt;li&gt;How often are collections created? Is there a pattern to their creation (eg. one every second, or a thousand once per minute)?&lt;/li&gt;
	&lt;li&gt;How are updates spread across the collections?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1238406" author="gregmurphy" created="Mon, 18 Apr 2016 11:14:43 +0000"  >&lt;p&gt;Report attached as requested. For the purpose of this test, we upgraded a single secondary to 3.2.5 again and ran perf against that secondary.&lt;/p&gt;</comment>
                            <comment id="1237588" author="thomas.schubert" created="Fri, 15 Apr 2016 20:17:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Can you please upload the perf report while you are experiencing this issue by following the steps below?&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;run the following command for a couple seconds:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;sudo perf record -a -g -s -p &amp;lt;pid&amp;gt; -o &amp;lt;output.data.filename&amp;gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;change the permissions of the outputted file:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;sudo chown &amp;lt;user&amp;gt; &amp;lt;output.data.filename&amp;gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;generate the report:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;perf report -i &amp;lt;output.data.filename&amp;gt; &amp;gt;&amp;gt; perf_report.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;attach perf_report.txt to this ticket.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1236956" author="gregmurphy" created="Fri, 15 Apr 2016 11:31:47 +0000"  >&lt;p&gt;Additionally, performance actually seems worse under 3.2.5 - initial analysis of the slow log is showing a more predictable mean of response time of very slow queries (&amp;gt;60s) but a significant increase in their frequency. In all the examples I&apos;ve looked at almost all of the response time is still waiting for the global lock.&lt;/p&gt;

&lt;p&gt;Interestingly, average checkpoint time appears to have gone down.&lt;/p&gt;

&lt;p&gt;We will revert to 3.2.4 to stabilise the system, but I will capture all of the diagnostic.data files from each node beforehand and upload them. Please let me know if there is any further debugging I can enable.&lt;/p&gt;</comment>
                            <comment id="1235782" author="gregmurphy" created="Thu, 14 Apr 2016 13:59:15 +0000"  >&lt;p&gt;OK - the file is uploading to the same secure URL you provided earlier in the ticket.&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;

&lt;p&gt;Greg&lt;/p&gt;</comment>
                            <comment id="1235767" author="thomas.schubert" created="Thu, 14 Apr 2016 13:43:04 +0000"  >&lt;p&gt;HI &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Sorry you are still experiencing issues, please upload the latest diagnostic.data and we will continue to investigate.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1235759" author="gregmurphy" created="Thu, 14 Apr 2016 13:36:26 +0000"  >&lt;p&gt;We upgraded one of the secondaries today, but unfortunately are still seeing the same behaviour on it. Examples:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Apr 14 13:19:07 SERVER mongod.27017[33669]: [conn11317] command DB.COLLECTION command: getMore { getMore: 21283805213298, collection: &quot;COLLECTION&quot; } cursorid:21283805213298 keyUpdates:0 writeConflicts:0 numYields:28 nreturned:3603 reslen:4195489 locks:{ Global: { acquireCount: { r: 58 }, acquireWaitCount: { r: 8 }, timeAcquiringMicros: { r: 50331051 } }, Database: { acquireCount: { r: 29 } }, Collection: { acquireCount: { r: 29 } } } protocol:op_query 50343ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Apr 14 13:19:06 SERVER mongod.27017[33669]: [conn12233] command DB.COLLECTION command: count { count: &quot;COLLECTION&quot;, query: { timeGroup: &quot;2016-04-12&quot; } } planSummary: COUNT_SCAN { timeGroup: -1 } keyUpdates:0 writeConflicts:0 numYields:59 reslen:62 locks:{ Global: { acquireCount: { r: 120 }, acquireWaitCount: { r: 3 }, timeAcquiringMicros: { r: 46770101 } }, Database: { acquireCount: { r: 60 } }, Collection: { acquireCount: { r: 60 } } } protocol:op_query 46772ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Apr 14 13:19:06 SERVER mongod.27017[33669]: [conn12256] command DB.COLLECTION command: find { find: &quot;COLLECTION&quot;, filter: { _id: &quot;20160414T1240-gsp-aeu000-mo04.gsp-aeu-1460637943500.gz&quot; }, limit: 1, singleBatch: true } planSummary: IDHACK keysExamined:0 docsExamined:0 idhack:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:143 locks:{ Global: { acquireCount: { r: 2 }, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 49799814 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 49938ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Would the updated diagnostic.data files help?&lt;/p&gt;</comment>
                            <comment id="1234305" author="gregmurphy" created="Wed, 13 Apr 2016 08:17:25 +0000"  >&lt;p&gt;Thanks - much appreciated - we&apos;ll upgrade as soon as the release is out and confirm.&lt;/p&gt;</comment>
                            <comment id="1233754" author="thomas.schubert" created="Tue, 12 Apr 2016 19:51:27 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I have examined the diagnostic data you have provided. You are correct that this issue is correlated to WiredTiger checkpoints. In particular, this issue may be related to a lock WiredTiger takes during checkpoints that will block workloads that create or drop collections or indexes. &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2346&quot; title=&quot;Don&amp;#39;t hold schema lock during checkpoint I/O&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2346&quot;&gt;&lt;del&gt;WT-2346&lt;/del&gt;&lt;/a&gt; improves this behavior, and will be included in the upcoming 3.2.5 release.&lt;/p&gt;

&lt;p&gt;Please upgrade to MongoDB 3.2.5 when it released and confirm if the issue is resolved.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1221199" author="gregmurphy" created="Thu, 31 Mar 2016 14:25:42 +0000"  >&lt;p&gt;Thanks Thomas, I&apos;ve just uploaded the archives from both secondaries.&lt;/p&gt;</comment>
                            <comment id="1221117" author="thomas.schubert" created="Thu, 31 Mar 2016 13:19:55 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I have created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/5e402889-5d24-4894-8820-30d36377cdd8.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; for you to use. Files you upload to this portal will only be visible to MongoDB staff investigating this ticket.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1220931" author="gregmurphy" created="Thu, 31 Mar 2016 08:17:41 +0000"  >&lt;p&gt;Thanks for the quick reply - can you give me a private upload link so that I don&apos;t have to attach all the data to the ticket please? (I&apos;ve got it ready to send when you do).&lt;/p&gt;</comment>
                            <comment id="1220414" author="ramon.fernandez" created="Wed, 30 Mar 2016 20:12:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregmurphy&quot; class=&quot;user-hover&quot; rel=&quot;gregmurphy&quot;&gt;gregmurphy&lt;/a&gt;, it is possible that the large number of files is indeed slowing things down. Can you please archive and upload the contents of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory for the affected secondaries? Seeing the full logs would also help determine if the issue is related to the number of files or if it&apos;s something else.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="197937">WT-1598</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="118112" name="perf_report.txt.gz" size="3549373" author="gregmurphy" created="Mon, 18 Apr 2016 11:14:43 +0000"/>
                            <attachment id="118086" name="s23433.png" size="75571" author="alexander.gorrod@mongodb.com" created="Mon, 18 Apr 2016 05:51:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000Y36sIIAR]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 30 Mar 2016 20:10:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 39 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 39 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>gregmurphy</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkc0n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsjwvz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrje1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>