<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:58:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-21754] CursorManager Lock is very hot</title>
                <link>https://jira.mongodb.org/browse/SERVER-21754</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Under heavy load the cursor manager lock causes negative scalability. &amp;#8211;&lt;br/&gt;
on power8 firestone (2 socket, 20 cores, 160 hwtreads) running redhat 7.2, mongo 3.2-pre, 4 10Gig E&apos;s hard wired to another firestone running 4 copies of ycsb with between 64 and 400 threads each. Throughput peaks at the lower end (64 threads each on ycsb) and has negative scaling with the larger numbers.  Tracing data and profile (from perf) information indicate that the bottleneck is on the cursor manager mutex lock. a prototype (read hack) replacing the mutex lock with a mongo spinlock increased peak performance from 240K ops to 320K ops and minimized the negative scaling&lt;/p&gt;</description>
                <environment></environment>
        <key id="242114">SERVER-21754</key>
            <summary>CursorManager Lock is very hot</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="charlie.swanson@mongodb.com">Charlie Swanson</assignee>
                                    <reporter username="jvf">Jim Van Fleet</reporter>
                        <labels>
                    </labels>
                <created>Thu, 3 Dec 2015 17:42:34 +0000</created>
                <updated>Wed, 6 Dec 2017 21:11:03 +0000</updated>
                            <resolved>Fri, 26 May 2017 14:31:30 +0000</resolved>
                                                    <fixVersion>3.5.8</fixVersion>
                                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>20</watches>
                                                                                                                <comments>
                            <comment id="1618705" author="charlie.swanson" created="Tue, 11 Jul 2017 13:41:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tdeneau&quot; class=&quot;user-hover&quot; rel=&quot;tdeneau&quot;&gt;tdeneau&lt;/a&gt; to specifically address your questions:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Are there any plans to address these contended lock issues, perhaps by additional partitioning?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Yes, that work will be tracked by the new ticket. I don&apos;t know exactly when it will happen, but you can watch it for updates! If you&apos;re particularly motivated, I would encourage you to try it yourself and open a pull request if it works!&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Also, are there any configuration options that would eliminate the need to go thru the code that calls these locks (on something like a ycsb db read-only benchmark)?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Not that I know of. We could conceivably add some sort of parameter to disable the top reporting (see &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/top/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the top command docs&lt;/a&gt;), but I don&apos;t think we have such a thing today.&lt;/p&gt;</comment>
                            <comment id="1618699" author="charlie.swanson" created="Tue, 11 Jul 2017 13:37:50 +0000"  >&lt;p&gt;Hi all,&lt;/p&gt;

&lt;p&gt;Thank you for all your work investigating. I&apos;ve opened a new ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30085&quot; title=&quot;Increase read scalability with many cores&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30085&quot;&gt;SERVER-30085&lt;/a&gt; which I&apos;ve linked to this one. Let&apos;s continue to track this investigation on that ticket, posting new findings there please. As I mentioned above, this ticket was scheduled mostly because we anticipated recent changes would increase contention on this already hot mutex. We did not actually solve the larger problem of increasing read scalability at high core counts though, so that work will be tracked by the new ticket, rather than re-opening this one.&lt;/p&gt;

&lt;p&gt;We really do appreciate all the investigation being done here, thanks for the help!&lt;/p&gt;

&lt;p&gt;Charlie&lt;/p&gt;</comment>
                            <comment id="1616832" author="tdeneau" created="Sat, 8 Jul 2017 00:10:15 +0000"  >&lt;p&gt;Charlie &amp;#8211;&lt;/p&gt;

&lt;p&gt;I have built from the tip of master getting&lt;br/&gt;
db version v3.5.8-487-gfa8c900&lt;br/&gt;
git version: fa8c900800a61cd399554f27d43ca58870a95187&lt;/p&gt;

&lt;p&gt;My problem:  I am still seeing poor scaling when trying to run the server on 64 cores (I am doing a read-only test, ycsb workloadc with as many ycsb clients running as needed to keep the server cores busy).&lt;/p&gt;

&lt;p&gt;When I run lttng traces with tracepoints in pthread_mutex_lock/unlock and kernel tracepoints in syscall_futex_enter/exit,I see that the previously hot Cursor lock, now partitioned is no longer an issue.  However, I am still seeing some pretty heavily contended locks.  I am not sure the best way to map these to mutexes in the source code while the server is busy (suggestions welcome), but using gdb, I saw that pthread_mutex_lock was called on these mutex addresses occasionally even when the server is &quot;quiet&quot;.&lt;/p&gt;

&lt;p&gt;From this, I can see that the busy locks are:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;the &quot;mutable SimpleMutex _lock&quot; in src/mongo/db/stats/top.h (called twice per db read op).&lt;/li&gt;
	&lt;li&gt;the  &quot;stdx::mutex _cacheLock&quot; in wiredtiger_session_cache.h, used for example in getSession and releaseSession.  Also called twice per db read op.  The time spent acquiring this lock was about 1/2 of that spent on the top.h lock&lt;/li&gt;
	&lt;li&gt;the stdx::mutex mutex in src/mongo/db/s/sharding_state.h.  Again called twice per db read op.&lt;br/&gt;
The time spent acquiring this lock was about 1/2 of that spent on the session_cache lock.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Are there any plans to address these contended lock issues, perhaps by additional partitioning?&lt;/p&gt;

&lt;p&gt;Also, are there any configuration options that would eliminate the need to go thru the code that calls these locks (on something like a ycsb db read-only benchmark)?&lt;/p&gt;
</comment>
                            <comment id="1609911" author="jvf" created="Wed, 28 Jun 2017 21:37:23 +0000"  >&lt;p&gt;Charlie &amp;#8211; I ran with your latest branch and collected performance data.  There is some cache line collisions in futexes, but on lightly used futexes. There may be other false sharing, but I can not directly detect that.  I made a quick hack to the definition of SimpleMutex to align the mutex on a cache line boundary and added padding to fill the cache line.  There was a very small improvement in overall performance and, of course, there were fewer obvious cache line overlaps in the futexes. Though it does not seem an issue now, it would probably be advisable to align mutexes/futexes to a processor dependent cache line size and pad to fill the cache line &amp;#8211; probably would not hurt to align and pad all mutexes ... In short, I do not think that there is a performance degradation because of the partitioned locks being in the same cache line.&lt;/p&gt;</comment>
                            <comment id="1607015" author="charlie.swanson" created="Mon, 26 Jun 2017 20:56:10 +0000"  >&lt;p&gt;Hi everyone,&lt;/p&gt;

&lt;p&gt;I just wanted to post an update as to the current status of this ticket.&lt;/p&gt;

&lt;p&gt;As a summary, my commit that went in to the 3.5.8 release attempted to address this problem by partitioning the lock into many locks, since we usually only need to interact with one entry in these maps/sets. There was some concern that doing so didn&apos;t actually help, or maybe actually hurt performance since the mutexes may lie on the same cache line - see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29462&quot; title=&quot;Eliminate possibility of false sharing in partitioned lock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29462&quot;&gt;&lt;del&gt;SERVER-29462&lt;/del&gt;&lt;/a&gt;. We have been unable to confirm or deny this hypothesis, but are experimenting with aligning the mutexes to a larger size to help avoid this problem. We haven&apos;t really seen any conclusive results so far, but continue to investigate. Part of the problem is that we have a lot of platforms to test on, and some experiments show speedups on some platforms and slowdowns on others.&lt;/p&gt;

&lt;p&gt;Here&apos;s &lt;a href=&quot;https://github.com/acmorrow/mongo/commits/partition-no-false-sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;a branch with some of the work we&apos;re evaluating&lt;/a&gt;. Any information you can provide about whether this approach works would be much appreciated.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tdeneau&quot; class=&quot;user-hover&quot; rel=&quot;tdeneau&quot;&gt;tdeneau&lt;/a&gt;, this change was motivated by recent work to move all aggregation cursors to the global cursor manager (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22541&quot; title=&quot;Aggregation plan executors should be owned by global cursor manager&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22541&quot;&gt;&lt;del&gt;SERVER-22541&lt;/del&gt;&lt;/a&gt;), so we anticipated this particular lock becoming a bottleneck for applications using many aggregations. There are no immediate plans to address scalability with other locks, but we have given a couple of them (such as the ones in &lt;tt&gt;mongo::ShardingState::getNS()&lt;/tt&gt; and &lt;tt&gt;Top&lt;/tt&gt;) some thought. The partitioned template seems like it could be applied to &lt;tt&gt;Top&lt;/tt&gt;, but it hasn&apos;t been tried. We have also looked into aligning various other mutexes in a similar way to see if that has any impact, again without conclusive results. There are no plans to add configuration options for this partitioning.&lt;/p&gt;</comment>
                            <comment id="1600991" author="tdeneau" created="Mon, 19 Jun 2017 18:29:59 +0000"  >&lt;p&gt;I am a latecomer to this issue but I am also interested in anything that helps scaling on hardware with lots of cores.  We also noticed the poor scaling with the CursorManager registerExecutor and deregisterExecutor.&lt;/p&gt;

&lt;p&gt;I can confirm that building from master helps the score in our read-only ycsb benchmark runs and that the Executors lock overhead is signficantly reduced.  But as was mentioned elsewhere there appear to be other scaling issues beyond the Executors lock which now dominate.&lt;/p&gt;

&lt;p&gt;Is the Partitioned template solution planned to be used for any other lock scaling issues?&lt;br/&gt;
Are there plans for any configuration options related to partitions?&lt;/p&gt;</comment>
                            <comment id="1597059" author="jvf" created="Wed, 14 Jun 2017 19:03:40 +0000"  >&lt;p&gt;Throughput on the master is now about the same as 3.4.  There is no cursor manager lock contention to speak of.&lt;/p&gt;

&lt;p&gt;This can be closed (if it is not already closed)&lt;/p&gt;</comment>
                            <comment id="1592246" author="jvf" created="Fri, 9 Jun 2017 14:47:12 +0000"  >&lt;p&gt;I incorrectly reported that the cursor manager patch decreased performance.  It is actually about the same as 3.5 base.  There is still a regression between 3.4 base and 3.5 base&lt;/p&gt;</comment>
                            <comment id="1587495" author="jvf" created="Mon, 5 Jun 2017 16:05:32 +0000"  >&lt;p&gt;changed some benchmark parameters (thread count) and I seem to have a stable 260K ops with the cursor manager change &amp;#8211; so the master w/cursor manager is about the same as base 3.5.&lt;/p&gt;

&lt;p&gt;locking differences between 3.5 and 3.5 master with cursor manager. Cursor manager change was successful in that it practically eliminated CursorManager lock.&lt;/p&gt;

&lt;p&gt;   0.02%     0.01%  conn39           libpthread-2.22.so   &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; pthread_mutex_unlock  3.5 locking&lt;font color=&quot;&quot;&gt;&lt;/font&gt; &lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;25.45%&lt;/del&gt;- mongo::ShardingState::getNS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;10.18%&lt;/del&gt;- mongo::CursorManager::deregisterExecutor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;9.41%&lt;/del&gt;- mongo::CursorManager::registerExecutor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;7.63%&lt;/del&gt;- mongo::Top::record&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;6.87%&lt;/del&gt;- mongo::LockManager::lock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;6.11%&lt;/del&gt;- mongo::LockManager::unlock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;5.85%&lt;/del&gt;- mongo::Top::incrementGlobalLatencyStats&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;5.85%&lt;/del&gt;- mongo::WiredTigerSessionCache::getSession&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;5.60%&lt;/del&gt;- mongo::DatabaseHolder::get&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;4.58%&lt;/del&gt;- mongo::WiredTigerSessionCache::releaseSession&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;3.05%&lt;/del&gt;- __wt_row_modify&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;2.29%&lt;/del&gt;- mongo::ScopedCollectionMetadata::_clear&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;2.04%&lt;/del&gt;- mongo::MetadataManager::getActiveMetadata&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;2.04%&lt;/del&gt;- mongo::(anonymous namespace)::threadStateChange&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.51%&lt;/del&gt;- mongo::repl::ReplicationCoordinatorImpl::getGetLastErrorDefault&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.51%&lt;/del&gt;- mongo::IndexCatalogEntryImpl::getMultikeyPaths&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.51%&lt;/del&gt;- mongo::KVCatalog::_findEntry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.51%&lt;/del&gt;- dl_iterate_phdr&lt;br/&gt;
                         -&lt;del&gt;1.02%&lt;/del&gt;- &lt;span class=&quot;error&quot;&gt;&amp;#91;...&amp;#93;&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;    0.02%     0.01%  conn101          libpthread-2.22.so   &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; pthread_mutex_lock 3.5 with cursor manager patch&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;30.21%&lt;/del&gt;- mongo::ShardingState::getNS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;12.81%&lt;/del&gt;- mongo::Top::incrementGlobalLatencyStats&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;11.44%&lt;/del&gt;- mongo::Top::record&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;5.26%&lt;/del&gt;- mongo::DatabaseHolderImpl::get&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;5.26%&lt;/del&gt;- mongo::WiredTigerSessionCache::getSession&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;4.35%&lt;/del&gt;- mongo::LockManager::lock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;3.66%&lt;/del&gt;- mongo::LockManager::unlock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;3.66%&lt;/del&gt;- mongo::WiredTigerSessionCache::releaseSession&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;3.43%&lt;/del&gt;- mongo::(anonymous namespace)::threadStateChange&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;3.20%&lt;/del&gt;- mongo::repl::ReplicationCoordinatorImpl::getGetLastErrorDefault&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;1.83%&lt;/del&gt;- mongo::CursorManager::deregisterExecutor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;1.83%&lt;/del&gt;- __wt_row_modify&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;1.60%&lt;/del&gt;- mongo::LockerImpl&amp;lt;false&amp;gt;::_unlockImpl&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;1.37%&lt;/del&gt;- mongo::CollectionShardingState::get&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;1.14%&lt;/del&gt;- mongo::assembleResponse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.92%&lt;/del&gt;- mongo::AutoStatsTracker::~AutoStatsTracker&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.92%&lt;/del&gt;- mongo::(anonymous namespace)::finishCurOp&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.92%&lt;/del&gt;- mongo::CursorManager::registerExecutor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.92%&lt;/del&gt;- pthread_mutex_lock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.92%&lt;/del&gt;- uw_frame_state_for&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.69%&lt;/del&gt;- mongo::MetadataManager::getActiveMetadata&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.69%&lt;/del&gt;- mongo::CollectionShardingState::getMetadata&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;del&gt;0.69%&lt;/del&gt;- mongo::fillOutPlannerParams&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;
</comment>
                            <comment id="1582760" author="jvf" created="Tue, 30 May 2017 15:33:03 +0000"  >&lt;p&gt;When I run performance tests with ycsb, 50% read/50% write, I get the &lt;br/&gt;
following nominal throughput  results:&lt;/p&gt;

&lt;p&gt;3.4 base &amp;#8211; about 280K ops&lt;br/&gt;
3.5 base &amp;#8211; about 253K ops&lt;br/&gt;
3.5 master - about 230K ops (with cursor manager change)&lt;/p&gt;

&lt;p&gt;I think my set-up is the same for each &amp;#8211; have one of two conclusions:&lt;br/&gt;
1) I am not running this correctly&lt;br/&gt;
2) both 3.5 and the cursor manager  have performance issues&lt;/p&gt;

&lt;p&gt;I  will be available to provide performance data ...&lt;/p&gt;

&lt;p&gt;Jim&lt;/p&gt;
</comment>
                            <comment id="1582759" author="jvf" created="Tue, 30 May 2017 15:31:29 +0000"  >&lt;p&gt;When I run performance tests with ycsb, 50% read/50% write, I get the following nominal throughput  results:&lt;/p&gt;

&lt;p&gt;3.4 base &amp;#8211; about 280K ops&lt;br/&gt;
3.5 base &amp;#8211; about 253K ops&lt;br/&gt;
3.5 master - about 230K ops (with cursor manager change)&lt;/p&gt;

&lt;p&gt;I think my set-up is the same for each &amp;#8211; have one of two conclusions:&lt;br/&gt;
1) I am not running this correctly &amp;#8211; perhaps some debug flags that I am not seeing??&lt;br/&gt;
2) both 3.5 and the cursor manager  have performance issues&lt;/p&gt;

&lt;p&gt;I  will be available to provide performance data ...&lt;/p&gt;

&lt;p&gt;Jim&lt;/p&gt;</comment>
                            <comment id="1580826" author="xgen-internal-githook" created="Fri, 26 May 2017 14:30:16 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;cswanson310&apos;, u&apos;name&apos;: u&apos;Charlie Swanson&apos;, u&apos;email&apos;: u&apos;charlie.swanson@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt; Partition CursorManager&apos;s structures&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fe2046a0eb715cec4d978454458f79b7588ee7a7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fe2046a0eb715cec4d978454458f79b7588ee7a7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1413651" author="david.storch" created="Thu, 20 Oct 2016 14:14:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Correct, this issue is currently on our backlog, meaning that it is on the roadmap but is not currently being worked on. We are not treating scalability improvements to the read path as a gating issue for the 3.4.0 release. Unfortunately, this means that strictly required work (e.g. ensuring a smooth user experience for the 3.2=&amp;gt;3.4 upgrade) has taken precedence. We are interested in making some related improvements to cursor management in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22541&quot; title=&quot;Aggregation plan executors should be owned by global cursor manager&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22541&quot;&gt;&lt;del&gt;SERVER-22541&lt;/del&gt;&lt;/a&gt;, so there is a decent chance that we will be looking at the scalability of the CursorManager in the coming months.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1410924" author="jvf" created="Mon, 17 Oct 2016 21:25:57 +0000"  >&lt;p&gt;I have not touched bases on this in a while &amp;#8211; what is the current status? (I see back log, but have you&apos;all made progress.)&lt;/p&gt;</comment>
                            <comment id="1297893" author="david.storch" created="Fri, 17 Jun 2016 15:13:37 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for providing the additional data. I believe your results demonstrate that the patch I provided earlier eliminates the contention issue within the CursorManager. However, it does clearly show that there is contention on the ShardingState lock. It appears that in order to improve the scalability of the read path, there may be several bottlenecks to eliminate, both in the query and sharding subsystems. Latching done in mongos or for recording diagnostic statistics on mongod could also potentially be problematic once the CursorManager and ShardingState bottlenecks are fixed. While we take scalability issues seriously, our team is currently focused on reaching feature completeness for 3.4 features. As time permits, we plan to revisit performance issues and do further performance testing post feature-complete. Thanks very much for your time and assistance running tests for this issue! For now I am putting this ticket back on the query team&apos;s backlog, but please do continue to watch for updates.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1293974" author="jvf" created="Tue, 14 Jun 2016 17:57:30 +0000"  >&lt;p&gt;perf annotate, report graph, report flat in attached gzip file - let me know if you want more&lt;/p&gt;</comment>
                            <comment id="1293726" author="david.storch" created="Tue, 14 Jun 2016 15:25:20 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Interesting, thanks for running the experiment. This suggests to me that we fixed the problem with CursorManager::_mutex, resulting in a new performance bottleneck while reading an in-memory data structure containing sharding-related metadata. I will have to investigate if there anything that can be done to reduce the amount of time spent in the ShardingState data structure&apos;s critical sections or if there is an opportunity to apply a similar partitioning strategy for synchronizing access to ShardingState. If you could share the profile data you collected, I would be interested in taking a look.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1292961" author="jvf" created="Mon, 13 Jun 2016 22:42:29 +0000"  >&lt;p&gt;Ran the tests yesterday and today.  full spreadsheet attached &lt;/p&gt;


&lt;p&gt;Essentially no change (slight improvement at 320 threads) &amp;#8211; better than base but still negative scaling.&lt;/p&gt;

&lt;p&gt;I also got some performance data &amp;#8211; profile and trace.  profile indicates about 30% of the cpu in raw_spin_lock.  perf graph snippit looks like:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                     |&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                     ---_raw_spin_lock&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |--52.52%-- futex_wait_setup&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |--67.12%-- __lll_lock_wait&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          pthread_mutex_lock&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |--77.74%-- mongo::ShardingState::getNS&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |          |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |          |--75.28%-- mongo::CollectionShardingState::get&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;and&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |--46.59%-- futex_wake&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |--55.12%-- pthread_mutex_unlock&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |          &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |--78.82%-- mongo::ShardingState::getNS&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                        |          |          |          |     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;     
&lt;p&gt;This, of course, indicates heavy contention for pthread_mutex_lock&lt;/p&gt;

&lt;p&gt;This was a test where ycsb had 240 threads &amp;#8211; vmstat shows an average of something like 100 threads and idle time of 35+%&lt;br/&gt;
I have not had a chance look at your code ...&lt;/p&gt;

&lt;p&gt;There is lots of data &amp;#8211; happy to share.&lt;/p&gt;
</comment>
                            <comment id="1290956" author="jvf" created="Fri, 10 Jun 2016 19:51:53 +0000"  >&lt;p&gt;yes, I can re-run the tests.  The machine is not available today, but I can probably get it on Monday&lt;/p&gt;</comment>
                            <comment id="1290736" author="david.storch" created="Fri, 10 Jun 2016 17:20:38 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks very much for doing the work to produce the perf numbers. Your results are pretty interesting. As a side note, could you please directly comment on this ticket for future communication about this issue? I have been alerted to the fact that messages to the JIRA email queue can sometimes get lost. This will also help to preserve context about our investigation for future reference and for other users who may be interested in the technical details of this issue.&lt;/p&gt;

&lt;p&gt;Based on the amount of concurrency in your test, it seems very plausible that the number of partitions in the initial prototype is far too low to make that much of a difference. I&apos;ve produced a second prototype build, very similar to the first, which has 256 partitions instead of 8. If 256 partitions doesn&apos;t do the trick, then we may have to re-evaluate our approach. The new build is available here:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/3ee21ae53b1aedf0820e627f37cf502871e7a0d2/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_3ee21ae53b1aedf0820e627f37cf502871e7a0d2_16_06_10_17_23_08.tgz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/3ee21ae53b1aedf0820e627f37cf502871e7a0d2/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_3ee21ae53b1aedf0820e627f37cf502871e7a0d2_16_06_10_17_23_08.tgz&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Would it be possible to rerun your tests? I have also attached file server21745_2.patch with the corresponding code changes, based on top of revision 3ee21ae53b1aedf0820e627f37cf502871e7a0d2 of the master branch.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1282892" author="david.storch" created="Thu, 2 Jun 2016 20:40:52 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Sure, no problem. I&apos;ve attached file &lt;tt&gt;server21754.patch&lt;/tt&gt; to this ticket. The patch applies cleanly as of revision 656480dfdb308884868a23ac41f9708b1b376431 of the master branch. Let me know if you need anything else from me for your testing.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1282862" author="jvf" created="Thu, 2 Jun 2016 20:16:27 +0000"  >&lt;p&gt;Hi, David &amp;#8211;&lt;/p&gt;

&lt;p&gt;Moving along on testing your change.  You did not send the source for the &lt;br/&gt;
change &amp;#8211; I could do a little better in analysis if I had the code. &lt;/p&gt;

&lt;p&gt;Jim &lt;/p&gt;



&lt;p&gt;From:   &quot;David Storch (JIRA)&quot; &amp;lt;jira@mongodb.org&amp;gt;&lt;br/&gt;
To:     Jim Van Fleet/Austin/Contr/IBM@IBMUS&lt;br/&gt;
Date:   05/19/2016 08:24 AM&lt;br/&gt;
Subject:        &lt;span class=&quot;error&quot;&gt;&amp;#91;MongoDB-JIRA&amp;#93;&lt;/span&gt; (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;) CursorManager Lock is very &lt;br/&gt;
hot&lt;/p&gt;




&lt;p&gt;     [ &lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel&lt;/a&gt; &lt;br/&gt;
]&lt;/p&gt;

&lt;p&gt;David Storch updated &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;:&lt;br/&gt;
----------------------------------&lt;br/&gt;
    Status: Waiting For User Input  (was: In Code Review)&lt;/p&gt;

&lt;p&gt;7.2, mongo 3.2-pre, 4 10Gig E&apos;s hard wired to another firestone running 4 &lt;br/&gt;
copies of ycsb with between 64 and 400 threads each. Throughput peaks at &lt;br/&gt;
the lower end (64 threads each on ycsb) and has negative scaling with the &lt;br/&gt;
larger numbers.  Tracing data and profile (from perf) information indicate &lt;br/&gt;
that the bottleneck is on the cursor manager mutex lock. a prototype (read &lt;br/&gt;
hack) replacing the mutex lock with a mongo spinlock increased peak &lt;br/&gt;
performance from 240K ops to 320K ops and minimized the negative scaling&lt;/p&gt;

&lt;p&gt;----------------------&lt;br/&gt;
This message was sent from MongoDB&apos;s issue tracking system. To respond to &lt;br/&gt;
this ticket, please login to &lt;a href=&quot;https://jira.mongodb.org&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org&lt;/a&gt; using your JIRA or &lt;br/&gt;
MMS credentials.&lt;/p&gt;




</comment>
                            <comment id="1273765" author="jvf" created="Tue, 24 May 2016 17:39:27 +0000"  >&lt;p&gt;Finally got the hardware &amp;#8211; getting it set up now &amp;#8211; progress when it &lt;br/&gt;
happens&lt;/p&gt;



&lt;p&gt;From:   &quot;David Storch (JIRA)&quot; &amp;lt;jira@mongodb.org&amp;gt;&lt;br/&gt;
To:     Jim Van Fleet/Austin/Contr/IBM@IBMUS&lt;br/&gt;
Date:   05/19/2016 08:24 AM&lt;br/&gt;
Subject:        &lt;span class=&quot;error&quot;&gt;&amp;#91;MongoDB-JIRA&amp;#93;&lt;/span&gt; (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;) CursorManager Lock is very &lt;br/&gt;
hot&lt;/p&gt;




&lt;p&gt;     [ &lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel&lt;/a&gt; &lt;br/&gt;
]&lt;/p&gt;

&lt;p&gt;David Storch updated &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;:&lt;br/&gt;
----------------------------------&lt;br/&gt;
    Status: Waiting For User Input  (was: In Code Review)&lt;/p&gt;

&lt;p&gt;7.2, mongo 3.2-pre, 4 10Gig E&apos;s hard wired to another firestone running 4 &lt;br/&gt;
copies of ycsb with between 64 and 400 threads each. Throughput peaks at &lt;br/&gt;
the lower end (64 threads each on ycsb) and has negative scaling with the &lt;br/&gt;
larger numbers.  Tracing data and profile (from perf) information indicate &lt;br/&gt;
that the bottleneck is on the cursor manager mutex lock. a prototype (read &lt;br/&gt;
hack) replacing the mutex lock with a mongo spinlock increased peak &lt;br/&gt;
performance from 240K ops to 320K ops and minimized the negative scaling&lt;/p&gt;

&lt;p&gt;----------------------&lt;br/&gt;
This message was sent from MongoDB&apos;s issue tracking system. To respond to &lt;br/&gt;
this ticket, please login to &lt;a href=&quot;https://jira.mongodb.org&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org&lt;/a&gt; using your JIRA or &lt;br/&gt;
MMS credentials.&lt;/p&gt;




</comment>
                            <comment id="1270807" author="jvf" created="Sat, 21 May 2016 00:51:27 +0000"  >&lt;p&gt;Hi &amp;#8211;&lt;/p&gt;

&lt;p&gt;I will have a projection of when I will get the machines on Monday.&lt;/p&gt;

&lt;p&gt;Jim&lt;/p&gt;



&lt;p&gt;From:   &quot;David Storch (JIRA)&quot; &amp;lt;jira@mongodb.org&amp;gt;&lt;br/&gt;
To:     Jim Van Fleet/Austin/Contr/IBM@IBMUS&lt;br/&gt;
Date:   05/19/2016 08:24 AM&lt;br/&gt;
Subject:        &lt;span class=&quot;error&quot;&gt;&amp;#91;MongoDB-JIRA&amp;#93;&lt;/span&gt; (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;) CursorManager Lock is very &lt;br/&gt;
hot&lt;/p&gt;




&lt;p&gt;    [ &lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;amp;focusedCommentId=1268729#comment-1268729&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;amp;focusedCommentId=1268729#comment-1268729&lt;/a&gt; &lt;br/&gt;
] &lt;/p&gt;

&lt;p&gt;David Storch commented on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;:&lt;br/&gt;
---------------------------------------&lt;/p&gt;

&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;, thanks for the update. There&apos;s no rush. I&apos;ll put this ticket in &lt;br/&gt;
a &quot;waiting&quot; state, pending the perf testing numbers.&lt;/p&gt;

&lt;p&gt;7.2, mongo 3.2-pre, 4 10Gig E&apos;s hard wired to another firestone running 4 &lt;br/&gt;
copies of ycsb with between 64 and 400 threads each. Throughput peaks at &lt;br/&gt;
the lower end (64 threads each on ycsb) and has negative scaling with the &lt;br/&gt;
larger numbers.  Tracing data and profile (from perf) information indicate &lt;br/&gt;
that the bottleneck is on the cursor manager mutex lock. a prototype (read &lt;br/&gt;
hack) replacing the mutex lock with a mongo spinlock increased peak &lt;br/&gt;
performance from 240K ops to 320K ops and minimized the negative scaling&lt;/p&gt;

&lt;p&gt;----------------------&lt;br/&gt;
This message was sent from MongoDB&apos;s issue tracking system. To respond to &lt;br/&gt;
this ticket, please login to &lt;a href=&quot;https://jira.mongodb.org&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org&lt;/a&gt; using your JIRA or &lt;br/&gt;
MMS credentials.&lt;/p&gt;




</comment>
                            <comment id="1268729" author="david.storch" created="Thu, 19 May 2016 13:23:13 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;, thanks for the update. There&apos;s no rush. I&apos;ll put this ticket in a &quot;waiting&quot; state, pending the perf testing numbers.&lt;/p&gt;</comment>
                            <comment id="1268249" author="jvf" created="Wed, 18 May 2016 21:32:27 +0000"  >&lt;p&gt;Hi Dave &amp;#8211;&lt;/p&gt;

&lt;p&gt;Right now, I do not have the configuration we did the original runs on ... &lt;br/&gt;
so I will have to scramble to find something suitable.&lt;/p&gt;

&lt;p&gt;So yes, I want to do this test ... but it will take me some time to get &lt;br/&gt;
and setup a machine. Actually, we used two before &amp;#8211; a driver and a &lt;br/&gt;
database machine. It may take me some time to get set up.  I will give you &lt;br/&gt;
a progress report by the end of the week.&lt;/p&gt;

&lt;p&gt;Jim&lt;/p&gt;



&lt;p&gt;From:   &quot;David Storch (JIRA)&quot; &amp;lt;jira@mongodb.org&amp;gt;&lt;br/&gt;
To:     Jim Van Fleet/Austin/Contr/IBM@IBMUS&lt;br/&gt;
Date:   05/18/2016 03:23 PM&lt;br/&gt;
Subject:        &lt;span class=&quot;error&quot;&gt;&amp;#91;MongoDB-JIRA&amp;#93;&lt;/span&gt; (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt;) CursorManager Lock is very &lt;br/&gt;
hot&lt;/p&gt;




&lt;p&gt;    [ &lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;amp;focusedCommentId=1267958#comment-1267958&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&amp;amp;focusedCommentId=1267958#comment-1267958&lt;/a&gt; &lt;br/&gt;
] &lt;/p&gt;

&lt;p&gt;David Storch edited comment on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21754&quot; title=&quot;CursorManager Lock is very hot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21754&quot;&gt;&lt;del&gt;SERVER-21754&lt;/del&gt;&lt;/a&gt; at 5/18/16 8:22 PM:&lt;br/&gt;
----------------------------------------------------------------&lt;/p&gt;

&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I have a proposed patch in code review which implements a partitioning &lt;br/&gt;
strategy in order to alleviate the CursorManager lock performance &lt;br/&gt;
bottleneck. On our performance testing infrastructure, the patch doesn&apos;t &lt;br/&gt;
show any major benefits, though I believe that the scaling problem is &lt;br/&gt;
likely be more severe on your hardware. Therefore, would you be willing to &lt;br/&gt;
test a build of this patch? The Enterprise RHEL PPC84LE binaries are &lt;br/&gt;
available for download here:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/f2f6163b0b26f1b18951c3d4d0f88a833c344523/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_f2f6163b0b26f1b18951c3d4d0f88a833c344523_16_05_18_18_38_39.tgz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/f2f6163b0b26f1b18951c3d4d0f88a833c344523/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_f2f6163b0b26f1b18951c3d4d0f88a833c344523_16_05_18_18_38_39.tgz&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;I&apos;d be very curious to see the results of running the same YSCB workload &lt;br/&gt;
against this build when you have them!&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;



&lt;p&gt;7.2, mongo 3.2-pre, 4 10Gig E&apos;s hard wired to another firestone running 4 &lt;br/&gt;
copies of ycsb with between 64 and 400 threads each. Throughput peaks at &lt;br/&gt;
the lower end (64 threads each on ycsb) and has negative scaling with the &lt;br/&gt;
larger numbers.  Tracing data and profile (from perf) information indicate &lt;br/&gt;
that the bottleneck is on the cursor manager mutex lock. a prototype (read &lt;br/&gt;
hack) replacing the mutex lock with a mongo spinlock increased peak &lt;br/&gt;
performance from 240K ops to 320K ops and minimized the negative scaling&lt;/p&gt;

&lt;p&gt;----------------------&lt;br/&gt;
This message was sent from MongoDB&apos;s issue tracking system. To respond to &lt;br/&gt;
this ticket, please login to &lt;a href=&quot;https://jira.mongodb.org&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org&lt;/a&gt; using your JIRA or &lt;br/&gt;
MMS credentials.&lt;/p&gt;




</comment>
                            <comment id="1267958" author="david.storch" created="Wed, 18 May 2016 18:27:42 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I have a proposed patch in code review which implements a partitioning strategy in order to alleviate the CursorManager lock performance bottleneck. On our performance testing infrastructure, the patch doesn&apos;t show any major benefits, though I believe that the scaling problem is likely be more severe on your hardware. Therefore, would you be willing to test a build of this patch? The Enterprise RHEL PPC84LE binaries are available for download here:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/f2f6163b0b26f1b18951c3d4d0f88a833c344523/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_f2f6163b0b26f1b18951c3d4d0f88a833c344523_16_05_18_18_38_39.tgz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://s3.amazonaws.com/mciuploads/mongodb-mongo-master/enterprise-rhel-71-ppc64le/f2f6163b0b26f1b18951c3d4d0f88a833c344523/binaries/mongo-mongodb_mongo_master_enterprise_rhel_71_ppc64le_f2f6163b0b26f1b18951c3d4d0f88a833c344523_16_05_18_18_38_39.tgz&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I&apos;d be very curious to see the results of running the same YSCB workload against this build when you have them!&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                            <comment id="1230486" author="james.wahlin@10gen.com" created="Fri, 8 Apr 2016 17:22:18 +0000"  >&lt;p&gt;Hi Jim,&lt;/p&gt;

&lt;p&gt;An assignee of &quot;Backlog&quot; just means that no one is &#8203;&lt;b&gt;actively&lt;/b&gt;&#8203; working on it at this point in time. Fix Version reflects our delivery target, which is currently during the 3.3 development cycle for release in MongoDB 3.4.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
James&lt;/p&gt;</comment>
                            <comment id="1230455" author="jvf" created="Fri, 8 Apr 2016 16:54:45 +0000"  >&lt;p&gt;Does &quot;Backlog&quot; mean this is no longer being worked on?&lt;/p&gt;</comment>
                            <comment id="1104583" author="david.storch" created="Thu, 3 Dec 2015 17:52:42 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jvf&quot; class=&quot;user-hover&quot; rel=&quot;jvf&quot;&gt;jvf&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the report. We actually noticed this during perf testing for the 3.2 release, although I believe this problem affects 3.0.x and 2.6.x versions as well. We have a few ideas for ways to fix this, so stay tuned. For the time being, I&apos;ve moved the ticket into &quot;Needs Triage&quot; state.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Dave&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="402506">SERVER-30085</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="390721">SERVER-29462</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="126151" name="mongoDBscaling.odp" size="40907" author="jvf" created="Mon, 13 Jun 2016 22:42:03 +0000"/>
                            <attachment id="126367" name="profile.tar.gz" size="11905052" author="jvf" created="Tue, 14 Jun 2016 17:57:19 +0000"/>
                            <attachment id="124556" name="server21754.patch" size="20695" author="david.storch@mongodb.com" created="Thu, 2 Jun 2016 20:40:52 +0000"/>
                            <attachment id="125802" name="server21754_2.patch" size="20929" author="david.storch@mongodb.com" created="Fri, 10 Jun 2016 17:28:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>30.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 3 Dec 2015 17:52:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 31 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-682</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 31 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>james.wahlin@mongodb.com</customfieldvalue>
            <customfieldvalue>jvf</customfieldvalue>
            <customfieldvalue>tdeneau</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkn2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrarwn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="898">Query 10 (02/22/16)</customfieldvalue>
    <customfieldvalue id="1040">Query 14 (05/13/16)</customfieldvalue>
    <customfieldvalue id="1084">Query 15 (06/03/16)</customfieldvalue>
    <customfieldvalue id="1571">Query 2017-03-27</customfieldvalue>
    <customfieldvalue id="1635">Query 2017-04-17</customfieldvalue>
    <customfieldvalue id="1671">Query 2017-05-08</customfieldvalue>
    <customfieldvalue id="1688">Query 2017-05-29</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrwhjj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>