<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:34:32 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-54805] Mongo become unresponsive, Spike in Connections and FD</title>
                <link>https://jira.mongodb.org/browse/SERVER-54805</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;System Configuration&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Os Version: CentOS 8&lt;br/&gt;
cat /etc/redhat-release&lt;br/&gt;
&lt;b&gt;CentOS Linux release 8.1.1911 (Core)&lt;/b&gt;&lt;br/&gt;
Mongo Version : 3.6.17&lt;br/&gt;
Storage Engine: mmapV1&lt;br/&gt;
Storage Type: Data Path in tmpfs&lt;/p&gt;

&lt;p&gt;Server Configuration:&lt;br/&gt;
 RAM: 160 GB&lt;br/&gt;
 HD: 100GB&lt;/p&gt;

&lt;p&gt;free -g&lt;br/&gt;
 total used free shared buff/cache available&lt;br/&gt;
Mem: 157 7 32 51 117 96&lt;br/&gt;
Swap: 3 0 3&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Problem&lt;/b&gt;&#160;&lt;b&gt;Description&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;Recently we upgraded from 3.6.9 to 3.6.17 as 3.6.17 is the release supports CentOS8.&#160;&#160; &#160; After this migration we are experiencing frequent issues where in after 1-2 days of system run, the mongo members are not responsive (MongoShell as well as Java API). So far we have found this is affected only on the secondary members. While debugging, we found that the lsof for hanging mongo process shows huge number around 32K +. The netstat doesnt provide the similar number but once it reaches around 35K it crashes. The RAM and CPU are not the bottleneck as we have enough free memory. We are also running the dbPath in tempfs.&lt;/p&gt;

&lt;p&gt;Based on the Production/Operation Checklist, ulimit also not a concern. But the &lt;b&gt;vm.max_map_count&lt;/b&gt; we set as &lt;b&gt;65530&lt;/b&gt; but, mongo recommends &lt;b&gt;128000&lt;/b&gt;. even if we increase the value, the replica member recovered immediately but the connection count and lsof is not reducing. We are just postponing the crash for may be another week. So we are not sure how this kernal parameter sould help.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;We have seen similar issues reported in the JIRA but due to unresponsiveness from submitter, the cases got closed.&#160;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46701&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-46701&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40625&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-40625&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;mongo logs shows&lt;/p&gt;

&lt;p&gt;&#160; 2021-02-24T22:50:02.603+0000 I -&#160; &#160; &#160; &#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable2021-02-24T22:50:02.603+0000 W EXECUTOR &lt;span class=&quot;error&quot;&gt;&amp;#91;conn480777&amp;#93;&lt;/span&gt; Terminating session due to error: InternalError: failed to create service entry worker thread2021-02-24T22:50:03.686+0000 I -&#160; &#160; &#160; &#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable2021-02-24T22:50:03.686+0000 W EXECUTOR &lt;span class=&quot;error&quot;&gt;&amp;#91;conn480778&amp;#93;&lt;/span&gt; Terminating session due to error: InternalError: failed to create service entry worker thread2021-02-24T22:50:03.690+0000 I -&#160; &#160; &#160; &#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable2021-02-24T22:50:03.690+0000 W EXECUTOR &lt;span class=&quot;error&quot;&gt;&amp;#91;conn480779&amp;#93;&lt;/span&gt; Terminating session due to error: InternalError: failed to create service entry worker thread2021-02-24T22:50:03.709+0000 I -&#160; &#160; &#160; &#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Attached the diagnostcs.data , systemctl output, rs.status(),rs.conf() and mongo logs.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1634472">SERVER-54805</key>
            <summary>Mongo become unresponsive, Spike in Connections and FD</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="lingzhi.deng@mongodb.com">Lingzhi Deng</assignee>
                                    <reporter username="veramasu@hcl.com">venkataramans rama</reporter>
                        <labels>
                    </labels>
                <created>Fri, 26 Feb 2021 03:33:19 +0000</created>
                <updated>Wed, 13 Oct 2021 18:16:32 +0000</updated>
                            <resolved>Thu, 24 Jun 2021 01:48:31 +0000</resolved>
                                    <version>3.6.17</version>
                    <version>4.0.23</version>
                                                    <component>MMAPv1</component>
                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="3895704" author="lingzhi.deng" created="Thu, 24 Jun 2021 01:48:31 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; was backported to 4.0, 4.2 and 4.4 and we believe that provides a workaround to the glibc bug. I am closing this ticket as a dup of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="3725356" author="JIRAUSER1257066" created="Mon, 19 Apr 2021 15:59:37 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for following up, and I understand the security implications that come with downgrading.&lt;/p&gt;

&lt;p&gt;While we can&apos;t provide a specific timeline of when this fix will be available, we encourage you to follow &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; where updates will be posted on the ticket as they happen.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3725337" author="JIRAUSER1258850" created="Mon, 19 Apr 2021 15:53:58 +0000"  >&lt;p&gt;Thank you Edwin for the confirmation. But unfortunately there is no lower version supported in glibc for centos8 and we can&apos;t downgrade to centos7 due to security vulnerabilities.&#160; May i know when the fix&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;&#160;will be available?&lt;/p&gt;</comment>
                            <comment id="3725284" author="JIRAUSER1257066" created="Mon, 19 Apr 2021 15:39:44 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I apologize for my delay in my follow up and greatly appreciate your patience. We suspect this behavior stems from a &lt;a href=&quot;https://sourceware.org/bugzilla/show_bug.cgi?id=25847&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;glibc bug in v2.27&lt;/a&gt; linked by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=neanton%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;neanton@gmail.com&quot;&gt;neanton@gmail.com&lt;/a&gt; above. You can find additional details about the effects of this bug on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;, which Andy has diligently documented, and our prospective solution for it.&lt;/p&gt;

&lt;p&gt;To work around this issue at this time, you may safely continue to restart any hung mongod instances. If possible, we recommend downgrading to a safe version of glibc (we believe to be a version older than v2.27) or upgrading when this particular bug is patched. You may also consider downgrading to CentOS 7. &lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3717998" author="schwerin" created="Wed, 14 Apr 2021 14:49:40 +0000"  >&lt;p&gt;After a fair amount of research, I&apos;ve filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; which describes how the glibc bug mentioned above almost certainly causes the behavior described in this ticket, and describing a modest change to mongodb code that should avoid this particular manifestation. Hopefully, the maintainers of the most common Linux distributions will get a chance to chance to patch the glibc bug before too long, as well.&lt;/p&gt;</comment>
                            <comment id="3714198" author="JIRAUSER1258850" created="Mon, 12 Apr 2021 22:03:13 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;Do you have any latest finding that you want to share with us?&lt;/p&gt;</comment>
                            <comment id="3703712" author="JIRAUSER1257066" created="Tue, 6 Apr 2021 17:58:50 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We highly appreciate you taking the time to upgrade to MongoDB 4.0! We&apos;re continuing our investigation on this issue and will update this ticket with our findings.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin &lt;/p&gt;</comment>
                            <comment id="3703290" author="JIRAUSER1258850" created="Tue, 6 Apr 2021 15:42:55 +0000"  >&lt;p&gt;Yes @Anton, we have been running fine with centOS 7 for years without any issues.&lt;/p&gt;</comment>
                            <comment id="3702440" author="neanton@gmail.com" created="Tue, 6 Apr 2021 09:06:44 +0000"  >&lt;p&gt;If that&#8217;s a glibc issue &lt;a href=&quot;https://sourceware.org/bugzilla/show_bug.cgi?id=25847&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://sourceware.org/bugzilla/show_bug.cgi?id=25847&lt;/a&gt; you can try downgrading to CentOS 7 and see if that helps. &lt;/p&gt;</comment>
                            <comment id="3700251" author="JIRAUSER1258850" created="Mon, 5 Apr 2021 04:06:16 +0000"  >&lt;p&gt;Hi Edwin and Team,&lt;/p&gt;

&lt;p&gt;As suggested we have migrated the mongo from 3.6.17 to 4.0.23 but we are observing the same issue in 4.0.23 as well.&#160; I&apos;ve uploaded 2 set of logs and from all 4 members of the RS. also the respective failed secondary GDB output as well. Could you please re-open the case?&lt;/p&gt;</comment>
                            <comment id="3682313" author="JIRAUSER1257066" created="Wed, 24 Mar 2021 19:29:52 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The ldd output that you provided doesn&apos;t make clear why we would be coming across this issue. Since we don&apos;t know exactly what&apos;s causing the oplog application thread to be stuck, and we don&apos;t have a working reproduction right now, we&apos;re not able to diagnose the issue. Given that MongoDB 3.6 is soon approaching EOL, we&apos;d like to see if this issue persists in MongoDB 4.0, so I recommend upgrading to MongoDB 4.0 first so we can see whether this issue persists in a later version. &lt;/p&gt;

&lt;p&gt;In the meantime, I will close this ticket as incomplete. If the issue persists when you are able to upgrade to MongoDB 4.0, please let us know and we will be happy to investigate further!&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3680719" author="JIRAUSER1258850" created="Wed, 24 Mar 2021 03:43:29 +0000"  >&lt;p&gt;Hi Edwin, Can you please comment on the ldd difference between 3.6.9 and 3.6.17?&lt;/p&gt;</comment>
                            <comment id="3679324" author="JIRAUSER1257066" created="Tue, 23 Mar 2021 15:06:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We don&apos;t know if this issue will persist when upgrading to 4.0, but you will be able to continue using MMAPv1 after the upgrade. If you choose to upgrade to 4.2 and later, you will need to change your storage engine to WiredTiger.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36516&quot; title=&quot;Secondary mongod becomes unresponsive occasionally&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36516&quot;&gt;&lt;del&gt;SERVER-36516&lt;/del&gt;&lt;/a&gt; was resolved after the reporter upgraded to 4.0 and switched their storage engine to WiredTiger, so I recommend at least attempting the upgrade to 4.0 first. I&apos;m unable to find any known issues with CentOS8 libraries on 3.6. We are interested in whether this issue persists in 4.0, and we look forward to hearing back when you&apos;re able to attempt the upgrade.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3678541" author="JIRAUSER1258850" created="Tue, 23 Mar 2021 06:22:44 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks again for your response. But for us, it would take time to migrate to WT. even if we upgrade to 4.0, we would like to keep the Storage Engine as mmapV1.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;So can you please let us know if the same issue we will hit in 4.0 with mmapV1?&lt;/li&gt;
	&lt;li&gt;Also as i said in my earlier problem statement, mongo 3.6.9 with CentOS7 has been running fine for a very long time. But only with the recent mongo 3.6.17 and centOS 8 combination we are observing this issue. We referred a similar issue reported in &lt;b&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36516&quot; title=&quot;Secondary mongod becomes unresponsive occasionally&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36516&quot;&gt;&lt;del&gt;SERVER-36516&lt;/del&gt;&lt;/a&gt;&lt;/b&gt; and tried the glibc upgrade as well. The system was running fine for 2 days and from 3rd day its started showing the same problem. So do you aware any known issues with CentOS8 and libraries that are used by mongod?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also one minor note on the dependent libraries difference between CentOS7 and 8. The following libraries are not shown in the ldd output of 3.6.17 in CentoS8. Will that make any difference?&lt;/p&gt;

&lt;p&gt;&lt;b&gt;ldd /usr/bin/mongod (3.6.17 in CentOS8)&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; linux-vdso.so.1 (0x00007ffde4188000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libresolv.so.2 =&amp;gt; /lib64/libresolv.so.2 (0x00007fcf497df000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libcrypto.so.1.1 =&amp;gt; /lib64/libcrypto.so.1.1 (0x00007fcf492f9000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libssl.so.1.1 =&amp;gt; /lib64/libssl.so.1.1 (0x00007fcf49065000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libdl.so.2 =&amp;gt; /lib64/libdl.so.2 (0x00007fcf48e61000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; librt.so.1 =&amp;gt; /lib64/librt.so.1 (0x00007fcf48c59000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libm.so.6 =&amp;gt; /lib64/libm.so.6 (0x00007fcf488d7000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libgcc_s.so.1 =&amp;gt; /lib64/libgcc_s.so.1 (0x00007fcf486bf000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libpthread.so.0 =&amp;gt; /lib64/libpthread.so.0 (0x00007fcf4849f000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libc.so.6 =&amp;gt; /lib64/libc.so.6 (0x00007fcf480dc000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; /lib64/ld-linux-x86-64.so.2 (0x00007fcf4cb36000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libz.so.1 =&amp;gt; /lib64/libz.so.1 (0x00007fcf47ec5000)&lt;/p&gt;

&lt;p&gt;&lt;b&gt;ldd /usr/bin/mongod (in 3.6.9 CentOS 7)&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; linux-vdso.so.1 =&amp;gt;&#160; (0x00007ffc636fd000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libresolv.so.2 =&amp;gt; /lib64/libresolv.so.2 (0x00007fafb7105000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libcrypto.so.10 =&amp;gt; /lib64/libcrypto.so.10 (0x00007fafb6ca3000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libssl.so.10 =&amp;gt; /lib64/libssl.so.10 (0x00007fafb6a31000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libdl.so.2 =&amp;gt; /lib64/libdl.so.2 (0x00007fafb682d000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; librt.so.1 =&amp;gt; /lib64/librt.so.1 (0x00007fafb6625000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libm.so.6 =&amp;gt; /lib64/libm.so.6 (0x00007fafb6323000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libgcc_s.so.1 =&amp;gt; /lib64/libgcc_s.so.1 (0x00007fafb610d000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libpthread.so.0 =&amp;gt; /lib64/libpthread.so.0 (0x00007fafb5ef1000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libc.so.6 =&amp;gt; /lib64/libc.so.6 (0x00007fafb5b24000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; /lib64/ld-linux-x86-64.so.2 (0x00007fafba42a000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libz.so.1 =&amp;gt; /lib64/libz.so.1 (0x00007fafb590e000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;libgssapi_krb5.so.2 =&amp;gt; /lib64/libgssapi_krb5.so.2 (0x00007fafb56c1000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libkrb5.so.3 =&amp;gt; /lib64/libkrb5.so.3 (0x00007fafb53d8000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libcom_err.so.2 =&amp;gt; /lib64/libcom_err.so.2 (0x00007fafb51d4000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libk5crypto.so.3 =&amp;gt; /lib64/libk5crypto.so.3 (0x00007fafb4fa1000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libkrb5support.so.0 =&amp;gt; /lib64/libkrb5support.so.0 (0x00007fafb4d91000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libkeyutils.so.1 =&amp;gt; /lib64/libkeyutils.so.1 (0x00007fafb4b8d000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libselinux.so.1 =&amp;gt; /lib64/libselinux.so.1 (0x00007fafb4966000)&lt;/p&gt;

&lt;p&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160; libpcre.so.1 =&amp;gt; /lib64/libpcre.so.1 (0x00007fafb4704000)&lt;/p&gt;</comment>
                            <comment id="3673610" author="JIRAUSER1257066" created="Fri, 19 Mar 2021 15:14:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;We identified the behavior that occurs, but we don&apos;t exactly know why it happens. The &lt;a href=&quot;https://docs.mongodb.com/manual/core/storage-engines/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MMAPv1 storage engine has been deprecated in favor of WiredTiger since MongoDB 4.0&lt;/a&gt;, which correlates with when this issue went away. You can find more information on the storage engine, &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/change-replica-set-wiredtiger/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;as well as migrating to WiredTiger from MMAPv1 as a replica set in our docs&lt;/a&gt;. We&apos;d love to hear back from you if this issue persists after you&apos;ve upgraded to MongoDB 4.0.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3670365" author="JIRAUSER1258850" created="Wed, 17 Mar 2021 23:40:15 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;&#160;Thank you for the analysis.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;We have been running 3.6.9 for long time and we never seen this issue reported. So Can you please let us know at what condition this issue can happen?&#160;&lt;/li&gt;
	&lt;li&gt;Also is there any workaround available?&lt;/li&gt;
	&lt;li&gt;Do we have any commands to unlock this replication lock if its stuck? This will help us to find mitigation plan if SEC becomes unresponsive.&lt;/li&gt;
	&lt;li&gt;Also are all 3.6 versions at risk or only from specific minor versions? This will help us for Production risk analysis.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Venkataraman&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3669233" author="JIRAUSER1257066" created="Wed, 17 Mar 2021 15:52:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your patience while we investigate this issue. After taking a look at the gdb dump, it appears many threads on the secondary nodes are stuck waiting for the main oplog application thread.&lt;/p&gt;

&lt;p&gt;The main oplog application is waiting in SyncTail::multiApply. &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Thread 37 (Thread 0x7f1917dbb700 (LWP 383262)):&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#0  0x00007f1c72d6447c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#1  0x00005594f9ecfddc in std::condition_variable::wait(std::unique_lock&amp;lt;std::mutex&amp;gt;&amp;amp;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#2  0x00005594f9565abb in mongo::ThreadPool::waitForIdle() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#3  0x00005594f890cb66 in mongo::repl::multiApply(mongo::OperationContext*, mongo::OldThreadPool*, std::vector&amp;lt;mongo::repl::OplogEntry, std::allocator&amp;lt;mongo::repl::OplogEntry&amp;gt; &amp;gt;, std::function&amp;lt;mongo::Status (std::vector&amp;lt;mongo::repl::OplogEntry const*, std::allocator&amp;lt;mongo::repl::OplogEntry const*&amp;gt; &amp;gt;*)&amp;gt;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#4  0x00005594f890d41d in mongo::repl::SyncTail::multiApply(mongo::OperationContext*, std::vector&amp;lt;mongo::repl::OplogEntry, std::allocator&amp;lt;mongo::repl::OplogEntry&amp;gt; &amp;gt;) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#5  0x00005594f890dcdc in mongo::repl::SyncTail::_oplogApplication(mongo::repl::ReplicationCoordinator*, mongo::repl::SyncTail::OpQueueBatcher*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#6  0x00005594f890e96b in mongo::repl::SyncTail::oplogApplication(mongo::repl::ReplicationCoordinator*) ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#7  0x00005594f888b1fc in mongo::repl::RSDataSync::_run() ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#8  0x00005594f9ed2e50 in execute_native_thread_routine ()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#9  0x00007f1c72d5e2de in start_thread () from /lib64/libpthread.so.0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;#10 0x00007f1c72a8f133 in clone () from /lib64/libc.so.6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;This means it has divided up a batch of oplog entries and given them to the pool of 16 parallel application threads, and is now waiting for that work to complete in ThreadPool::waitForIdle. However none of those 16 worker threads is active, so they should have signaled the main oplog application thread that they had finished their work, but for some reason didn&apos;t.&lt;/p&gt;

&lt;p&gt;This issue has been identified in the past but we haven&apos;t resolved it. In more recent versions, this issue seems to have gone away. Since MongoDB version 3.6 is scheduled to go EOL next month as stated in our &lt;a href=&quot;https://www.mongodb.com/support-policy&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support policy&lt;/a&gt;, can you upgrade to 4.0 to see if this problem persists?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3663658" author="JIRAUSER1258850" created="Mon, 15 Mar 2021 07:47:48 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;While I wait for your investigation, I see the following observation from currentOp whenever we hit the issue. The&#160;rsSync in currentOp shows Global W lock and may be it stuck at the lock. All the other operations including listDatabase are waiting for the lock.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;listDatabase in currentOp&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;{ { &quot;host&quot; : &quot;SM10:27717&quot;, &quot;desc&quot; : &quot;conn204498&quot;, &quot;connectionId&quot; : 204498, &quot;client&quot; : &quot;127.0.0.1:59912&quot;, &quot;appName&quot; : &quot;MongoDB Shell&quot;, &quot;clientMetadata&quot; : { &quot;application&quot; :&lt;/p&gt;

{ &quot;name&quot; : &quot;MongoDB Shell&quot; }

&lt;p&gt;, &quot;driver&quot; : { &quot;name&quot; : &quot;MongoDB Internal Client&quot;, &quot;version&quot; : &quot;3.6.17&quot; }, &quot;os&quot; : { &quot;type&quot; : &quot;Linux&quot;, &quot;name&quot; : &quot;CentOS Linux release 8.1.1911 (Core) &quot;, &quot;architecture&quot; : &quot;x86_64&quot;, &quot;version&quot; : &quot;Kernel 4.18.0-193.14.2.el8_2.x86_64&quot; } }, &quot;active&quot; : true, &quot;currentOpTime&quot; : &quot;2021-03-11T00:24:12.761+0000&quot;, &quot;opid&quot; : 736216208, &quot;killPending&quot; : true, &quot;secs_running&quot; : NumberLong(6535), &quot;microsecs_running&quot; : NumberLong(&quot;6535554063&quot;), &quot;op&quot; : &quot;command&quot;, &quot;ns&quot; : &quot;admin.$cmd&quot;, &quot;command&quot; : { &quot;listDatabases&quot; : 1, &quot;lsid&quot; :&lt;/p&gt;

{ &quot;id&quot; : UUID(&quot;f3d38946-56de-497c-bacb-f8cf77357ca8&quot;) }

&lt;p&gt;, &quot;$readPreference&quot; : { &quot;mode&quot; : &quot;secondaryPreferred&quot; }, &quot;$db&quot; : &quot;admin&quot; }, &quot;numYields&quot; : 0, &quot;locks&quot; : { &quot;Global&quot; : &quot;r&quot; }, &quot;waitingForLock&quot; : true, &quot;lockStats&quot; : { &quot;Global&quot; : { &quot;acquireCount&quot; :&lt;/p&gt;

{ &quot;r&quot; : NumberLong(1) }

&lt;p&gt;, &quot;acquireWaitCount&quot; : { &quot;r&quot; : NumberLong(1) }, &quot;timeAcquiringMicros&quot; : { &quot;r&quot; : NumberLong(&quot;6535434929&quot;) } } } },&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;rsSync in currentOp&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&quot;host&quot; : &quot;SM10:27717&quot;, &quot;host&quot; : &quot;SM10:27717&quot;, &quot;desc&quot; : &quot;rsSync&quot;, &quot;active&quot; : true, &quot;currentOpTime&quot; : &quot;2021-03-11T00:24:12.760+0000&quot;, &quot;opid&quot; : 735821910, &quot;op&quot; : &quot;none&quot;, &quot;ns&quot; : &quot;&quot;, &quot;command&quot; : { }, &quot;numYields&quot; : 0, &quot;locks&quot; : { &quot;Global&quot; : &quot;W&quot; }, &quot;waitingForLock&quot; : false, &quot;lockStats&quot; : { &quot;Global&quot; : { &quot;acquireCount&quot; :&lt;/p&gt;

{ &quot;r&quot; : NumberLong(3), &quot;w&quot; : NumberLong(3), &quot;W&quot; : NumberLong(1) }

&lt;p&gt;}, &quot;MMAPV1Journal&quot; : { &quot;acquireCount&quot; :&lt;/p&gt;

{ &quot;r&quot; : NumberLong(2), &quot;w&quot; : NumberLong(6) }

&lt;p&gt;}, &quot;Database&quot; : { &quot;acquireCount&quot; :&lt;/p&gt;

{ &quot;r&quot; : NumberLong(2), &quot;w&quot; : NumberLong(3) }

&lt;p&gt;}, &quot;Collection&quot; : { &quot;acquireCount&quot; :&lt;/p&gt;

{ &quot;R&quot; : NumberLong(1), &quot;W&quot; : NumberLong(3) }

&lt;p&gt;} } },&lt;/p&gt;</comment>
                            <comment id="3647461" author="JIRAUSER1257066" created="Thu, 4 Mar 2021 18:48:26 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I appreciate your patience while we investigate this issue. We are still reviewing the diagnostics that you&apos;ve provided us and will post updates to this ticket as we further understand this issue.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="3647096" author="JIRAUSER1258850" created="Thu, 4 Mar 2021 16:47:41 +0000"  >&lt;p&gt;Hi Edwin,&lt;br/&gt;
Do you have any update on the finding from gdb?&lt;/p&gt;</comment>
                            <comment id="3642912" author="JIRAUSER1258850" created="Tue, 2 Mar 2021 19:56:54 +0000"  >&lt;p&gt;Thank you for setting up the secure portal. I have uploaded GDB output from 2 members.&lt;/p&gt;</comment>
                            <comment id="3641903" author="dmitry.agranat" created="Tue, 2 Mar 2021 14:45:15 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;, you can upload everything into &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/43d80394-8aef-49dc-a784-d61723228370.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this secure portal&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3641358" author="JIRAUSER1258850" created="Tue, 2 Mar 2021 06:49:33 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;Thanks for your initial analysis and response.&lt;/p&gt;

&lt;p&gt;FYI, we have quickly tried with the latest mongo 3.6.22 as well and we run into the same issue.&#160;the GDB dump and logs are around 250MB. Is there any other place where I can upload such huge logs?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Venkataraman&lt;/p&gt;</comment>
                            <comment id="3640161" author="JIRAUSER1257066" created="Mon, 1 Mar 2021 18:11:07 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for reporting this issue and providing logs and FTDC. It really helps with beginning this investigation.&lt;/p&gt;

&lt;p&gt;To help identify why a particular node is unresponsive, would you please collect gdb from the unresponsive node the next time this issue occurs and upload the file?&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gdb -p $(pidof mongod) -batch -ex &apos;thread apply all bt&apos; &amp;gt; gdb_`date +&quot;%Y-%m-%d_%H-%M-%S&quot;`.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1678281">SERVER-56054</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1678281">SERVER-56054</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1881608">SERVER-60164</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="301966" name="LOGS.tar.gz" size="24562747" author="veramasu@hcl.com" created="Fri, 26 Feb 2021 02:22:59 +0000"/>
                            <attachment id="301965" name="diagnostic_data.tar.gz" size="60715754" author="veramasu@hcl.com" created="Fri, 26 Feb 2021 02:27:20 +0000"/>
                            <attachment id="302912" name="mongo-9vb-27767.png" size="180791" author="edwin.zhou@mongodb.com" created="Wed, 3 Mar 2021 22:06:50 +0000"/>
                            <attachment id="302913" name="mongo-GDB-10va-27737.png" size="173813" author="edwin.zhou@mongodb.com" created="Wed, 3 Mar 2021 22:06:50 +0000"/>
                            <attachment id="301967" name="mongo-jira.txt" size="28481" author="veramasu@hcl.com" created="Fri, 26 Feb 2021 02:21:46 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>24.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 1 Mar 2021 18:11:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 33 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-56054'>SERVER-56054</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 33 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>neanton@gmail.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>lingzhi.deng@mongodb.com</customfieldvalue>
            <customfieldvalue>veramasu@hcl.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyx0d3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hz2u0n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5035">Repl 2021-06-14</customfieldvalue>
    <customfieldvalue id="5036">Repl 2021-06-28</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hywmm7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>