<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:41:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16562] Extremely slow shutdown for WiredTiger (non-LSM)</title>
                <link>https://jira.mongodb.org/browse/SERVER-16562</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I loaded 2B docs via the insert benchmark to create a 300G database last night. The load stopped more than 8 hours ago. This morning I did &quot;kill $( pidof mongod )&quot; to shutdown mongod and the shutdown took 12 minutes. &lt;/p&gt;

&lt;p&gt;This reproduces with the default (non-LSM) engine for WiredTiger. This does not reproduce with the LSM engine. This is the thread stack for the thread doing all of the work at shutdown. My host has very fast PCIe-flash for storage.&lt;/p&gt;

&lt;p&gt; Looks like a long checkpoint to me:&lt;br/&gt;
_&lt;em&gt;free_update_list,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;

&lt;p&gt;During the shutdown this error message was printed a few times:&lt;/p&gt;

&lt;p&gt;2014-12-16T07:59:02.232-0800 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;TTLMonitor&amp;#93;&lt;/span&gt; LockerId 411893 has been waiting to acquire lock for more than 30 seconds. MongoDB will print the lock manager state and the stack of the thread that&lt;br/&gt;
 has been waiting, for diagnostic purposes. This message does not necessary imply that the server is experiencing an outage, but might be an indication of an overload.&lt;br/&gt;
2014-12-16T07:59:02.232-0800 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;TTLMonitor&amp;#93;&lt;/span&gt; Dumping LockManager @ 0x1ad5d20&lt;br/&gt;
2014-12-16T07:59:02.232-0800 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;TTLMonitor&amp;#93;&lt;/span&gt; Lock @ 0x2903580: &lt;/p&gt;
{2305843009213693953: Global, 1}
&lt;p&gt;GRANTED:&lt;br/&gt;
        LockRequest 0 @ 0x7F9C8B743350: Mode = X; ConvertMode = NONE; EnqueueAtFront = 1; CompatibleFirst = 1; &lt;/p&gt;

&lt;p&gt;PENDING:&lt;br/&gt;
        LockRequest 411893 @ 0x2BC8A00: Mode = IS; ConvertMode = NONE; EnqueueAtFront = 0; CompatibleFirst = 0; &lt;br/&gt;
2014-12-16T07:59:02.232-0800 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;TTLMonitor&amp;#93;&lt;/span&gt; &lt;br/&gt;
2014-12-16T07:59:02.235-0800 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;TTLMonitor&amp;#93;&lt;/span&gt; &lt;br/&gt;
 0xf06559 0x99c74a 0x9a41cc 0x99d44b 0x993520 0x994ed0 0xd40012 0xd41392 0xe98ec0 0xf528f4 0x7f9c92c48fa8 0x7f9c91d2a5ad&lt;/p&gt;</description>
                <environment></environment>
        <key id="174792">SERVER-16562</key>
            <summary>Extremely slow shutdown for WiredTiger (non-LSM)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.benvenuto@mongodb.com">Mark Benvenuto</assignee>
                                    <reporter username="mdcallag">Mark Callaghan</reporter>
                        <labels>
                    </labels>
                <created>Tue, 16 Dec 2014 16:17:06 +0000</created>
                <updated>Wed, 24 Dec 2014 17:21:38 +0000</updated>
                            <resolved>Thu, 18 Dec 2014 02:15:37 +0000</resolved>
                                    <version>2.8.0-rc2</version>
                                    <fixVersion>2.8.0-rc4</fixVersion>
                                    <component>Concurrency</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="789190" author="xgen-internal-githook" created="Thu, 18 Dec 2014 02:11:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16562&quot; title=&quot;Extremely slow shutdown for WiredTiger (non-LSM)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16562&quot;&gt;&lt;del&gt;SERVER-16562&lt;/del&gt;&lt;/a&gt;: Do fast wiredtiger shutdown&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/46589450c11ce5cc8cfef2eaf47342d53b271a35&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/46589450c11ce5cc8cfef2eaf47342d53b271a35&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="788975" author="asya" created="Wed, 17 Dec 2014 22:13:18 +0000"  >&lt;p&gt;I&apos;ve seen slow shutdown as well - I&apos;d assumed it was the last checkpoint and in fact if I sleep for 60 to 120 seconds before shutting down it takes less time.  Could be both - last checkpoint and freeing memory?&lt;/p&gt;</comment>
                            <comment id="788432" author="mdcallag" created="Wed, 17 Dec 2014 17:21:23 +0000"  >&lt;p&gt;I wonder if tuning tcmalloc can help here. Back in the day when I used it with MySQL we reduced the amount of memory it would cache per thread to avoid wasting memory for workloads with 1000+ threads. &lt;/p&gt;

&lt;p&gt;I repeated the shutdown test after for jemalloc. I didn&apos;t not try to tune it. The shutdown took ~17 minutes. And from one sample thread stack there is some time in jemalloc code:&lt;br/&gt;
jemalloc_je_arena_dalloc_bin_locked,jemalloc_je_tcache_bin_flush_small,jemalloc_je_tcache_dalloc_small,jemalloc_je_arena_dalloc,jemalloc_je_idalloct,jemalloc_je_iqalloct,jemalloc_je_iqalloc,ifree,free,_&lt;em&gt;wt_free_int,&lt;/em&gt;&lt;em&gt;free_update_list,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;

&lt;p&gt;And the top N functions from linux perf during a 10 second sample&lt;/p&gt;

&lt;p&gt;    34.07%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __wt_page_out                        &lt;br/&gt;
    29.32%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; jemalloc_je_tcache_bin_flush_small   &lt;br/&gt;
    17.26%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; free                                 &lt;br/&gt;
    17.13%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; jemalloc_je_arena_dalloc_bin_locked  &lt;br/&gt;
     0.55%   mongod  libpthread-2.17.so  &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; pthread_mutex_lock                   &lt;br/&gt;
     0.40%   mongod  libpthread-2.17.so  &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; pthread_mutex_unlock                 &lt;br/&gt;
     0.23%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __wt_free_int                        &lt;br/&gt;
     0.14%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; arena_purge                          &lt;br/&gt;
     0.14%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; arena_run_tree_insert                &lt;br/&gt;
     0.06%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;   &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; futex_wait                           &lt;br/&gt;
     0.06%   mongod  libc-2.17.so        &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __memmove_ssse3_back                 &lt;br/&gt;
     0.05%   mongod  libpthread-2.17.so  &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; pthread_cond_timedwait@@GLIBC_2.3.2  &lt;br/&gt;
     0.05%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; arena_avail_tree_insert              &lt;br/&gt;
     0.04%   mongod  mongod              &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; arena_chunk_dirty_remove             &lt;br/&gt;
     0.03%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;   &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; _raw_spin_lock                       &lt;/p&gt;</comment>
                            <comment id="787729" author="mjc@wiredtiger.com" created="Tue, 16 Dec 2014 23:01:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mdcallag&quot; class=&quot;user-hover&quot; rel=&quot;mdcallag&quot;&gt;mdcallag&lt;/a&gt;, what you&apos;re seeing is WiredTiger freeing memory that was allocated to cache data.  There shouldn&apos;t be any I/O involved because periodic checkpoints will ensure that the cache is clean.&lt;/p&gt;

&lt;p&gt;MongoDB defaults to allowing the WiredTiger cache to grow to half the size of system RAM, so there could be 100GB of memory to free, depending on your test system.&lt;/p&gt;

&lt;p&gt;You may recall that you suggested we add a way for WiredTiger to shut down without freeing that memory, so there is a &lt;tt&gt;leak_memory&lt;/tt&gt; configuration to &lt;tt&gt;WT_CONNECTION::close&lt;/tt&gt; that would avoid all of this work.  Of course, the downside of using that is that leak checking tools will report leaked memory.  It is not currently used or exposed via MongoDB: we&apos;ll have a discussion about whether it can be turned on.&lt;/p&gt;</comment>
                            <comment id="787327" author="mdcallag" created="Tue, 16 Dec 2014 18:40:23 +0000"  >&lt;p&gt;Tried hierarchical &quot;perf record -g&quot; but (nil) was used rather than names for calling functions so the result is garbage&lt;/p&gt;</comment>
                            <comment id="787324" author="mdcallag" created="Tue, 16 Dec 2014 18:39:17 +0000"  >&lt;p&gt;Top-N from &quot;perf record&quot; with flat profiling&lt;/p&gt;

&lt;p&gt;    39.87%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __wt_page_out&lt;br/&gt;
    29.48%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; tc_free&lt;br/&gt;
    14.09%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; tcmalloc::CentralFreeList::ReleaseToSpans(void*)&lt;br/&gt;
     8.00%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)&lt;br/&gt;
     7.61%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; tcmalloc::CentralFreeList::ReleaseListToSpans(void*)&lt;br/&gt;
     0.26%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __wt_free_int&lt;br/&gt;
     0.11%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; update_sd_lb_stats&lt;br/&gt;
     0.06%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; __wt_epoch&lt;br/&gt;
     0.05%   mongod  mongod             &lt;span class=&quot;error&quot;&gt;&amp;#91;.&amp;#93;&lt;/span&gt; tcmalloc::CentralFreeList::InsertRange(void*, void*, int)&lt;br/&gt;
     0.04%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; rcu_check_callbacks&lt;br/&gt;
     0.03%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; _raw_spin_lock&lt;br/&gt;
     0.03%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; do_timer&lt;br/&gt;
     0.03%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; cpuacct_charge&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; _raw_spin_lock_irqsave&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; apic_timer_interrupt&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; smp_apic_timer_interrupt&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; update_cfs_shares&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; perf_event_task_tick&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; lapic_next_deadline&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; __do_softirq&lt;br/&gt;
     0.02%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; trigger_load_balance&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; ktime_get_update_offsets&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; native_sched_clock&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; scheduler_tick&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; irq_enter&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; ktime_get&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; rcu_irq_exit&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; x86_pmu_disable&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; update_context_time.isra.48&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; kthread_data&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; calc_global_load&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; __irq_work_run&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; hrtimer_interrupt&lt;br/&gt;
     0.01%   mongod  &lt;span class=&quot;error&quot;&gt;&amp;#91;kernel.kallsyms&amp;#93;&lt;/span&gt;  &lt;span class=&quot;error&quot;&gt;&amp;#91;k&amp;#93;&lt;/span&gt; rcu_eqs_enter_common.isra.54&lt;/p&gt;</comment>
                            <comment id="787321" author="mdcallag" created="Tue, 16 Dec 2014 18:38:07 +0000"  >&lt;p&gt;Repeated test using fsync on commit and then grabbed thread stacks on shutdown. Here is a sample...&lt;/p&gt;

&lt;p&gt;SLL_PopRange,PopRange,tcmalloc::ThreadCache::ReleaseToCentralCache,tcmalloc::ThreadCache::ListTooLong,Deallocate,do_free_helper,do_free_with_callback,do_free,tc_free,_&lt;em&gt;wt_free_int,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;

&lt;p&gt;_&lt;em&gt;free_update_list,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;

&lt;p&gt;tcmalloc::CentralFreeList::ReleaseToSpans,tcmalloc::CentralFreeList::ReleaseListToSpans,tcmalloc::CentralFreeList::InsertRange,tcmalloc::ThreadCache::ReleaseToCentralCache,tcmalloc::ThreadCache::ListTooLong,Deallocate,do_free_helper,do_free_with_callback,do_free,tc_free,_&lt;em&gt;wt_free_int,&lt;/em&gt;&lt;em&gt;free_update_list,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;

&lt;p&gt;_&lt;em&gt;free_update_list,&lt;/em&gt;&lt;em&gt;free_skip_list,&lt;/em&gt;&lt;em&gt;free_skip_array,&lt;/em&gt;&lt;em&gt;free_page_row_leaf,&lt;/em&gt;&lt;em&gt;wt_page_out,&lt;/em&gt;&lt;em&gt;wt_ref_out,&lt;/em&gt;&lt;em&gt;wt_rec_page_clean_update,&lt;/em&gt;&lt;em&gt;evict_file,&lt;/em&gt;&lt;em&gt;wt_cache_op,&lt;/em&gt;&lt;em&gt;wt_checkpoint_close,&lt;/em&gt;&lt;em&gt;wt_conn_btree_sync_and_close,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard_single,&lt;/em&gt;&lt;em&gt;wt_conn_dhandle_discard,&lt;/em&gt;&lt;em&gt;wt_connection_close,&lt;/em&gt;_conn_close,mongo::WiredTigerKVEngine::cleanShutdown,shutdownServer,mongo::exitCleanly,mongo::(anonymous,boost::(anonymous,start_thread,clone&lt;/p&gt;</comment>
                            <comment id="787117" author="mdcallag" created="Tue, 16 Dec 2014 16:44:17 +0000"  >&lt;p&gt;When this command line is used to get LSM from WT, then the shutdown is fast (10 seconds):&lt;br/&gt;
 bin.tcm/mongod --config /data/mysql/mongo/mongo.conf --storageEngine wiredTiger --wiredTigerIndexConfig=&quot;type=lsm&quot;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 16 Dec 2014 23:01:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 9 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ernie.hershey@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 9 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>mdcallag</customfieldvalue>
            <customfieldvalue>mjc@wiredtiger.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlgkn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs4ttb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>153168</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;command line for starting mongod:&lt;br/&gt;
bin.tcm/mongod --config /data/mysql/mongo/mongo.conf --storageEngine wiredTiger&lt;/p&gt;

&lt;p&gt;configuration file:&lt;br/&gt;
processManagement:&lt;br/&gt;
  fork: true&lt;br/&gt;
systemLog:&lt;br/&gt;
  destination: file&lt;br/&gt;
  path: /data/mysql/mongo/log&lt;br/&gt;
  logAppend: true&lt;br/&gt;
storage:&lt;br/&gt;
  dbPath: /data/mysql/mongo/data&lt;br/&gt;
  journal:&lt;br/&gt;
    enabled: true&lt;br/&gt;
  mmapv1:&lt;br/&gt;
    syncPeriodSecs: 60&lt;br/&gt;
    journal:&lt;br/&gt;
      commitIntervalMs: 100&lt;/p&gt;

&lt;p&gt;Test case is the insert benchmark changed to insert 2B rows (200M rows per thread, 10 insert threads) with a 2000/byte char field.&lt;br/&gt;
export MAX_ROWS=200000000&lt;br/&gt;
export LENGTH_CHAR_FIELDS=2000&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hriffz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>