<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:57:42 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-63402] High query response time for find operation in mongo 4.0.27 with mmap storage engine with random intervals (5/7/12/20 hours)</title>
                <link>https://jira.mongodb.org/browse/SERVER-63402</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Scenario&lt;/b&gt;&#160;- We have application which is doing CRUD operations on database. From past 2 years we are using mongo 3.6.9 and system is working fine without any issues but when we upgraded the database from 3.6.9 to 4.0.27 and then we started getting high query response time for find operation but it is not consistent in nature ,sometimes we are getting high query response in 5 hours or 7 hours or even 11 hours.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1979227">SERVER-63402</key>
            <summary>High query response time for find operation in mongo 4.0.27 with mmap storage engine with random intervals (5/7/12/20 hours)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="edwin.zhou@mongodb.com">Edwin Zhou</assignee>
                                    <reporter username="kg3634@gmail.com">KAPIL GUPTA</reporter>
                        <labels>
                    </labels>
                <created>Tue, 8 Feb 2022 11:29:55 +0000</created>
                <updated>Mon, 28 Feb 2022 16:00:37 +0000</updated>
                            <resolved>Mon, 28 Feb 2022 16:00:36 +0000</resolved>
                                    <version>4.0.27</version>
                                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="4380607" author="JIRAUSER1257066" created="Mon, 28 Feb 2022 16:00:37 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for following up that you&apos;ve experience this issue more frequently after applying the patch. &lt;/p&gt;

&lt;p&gt;I will now close this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; since it works around the glibc 2.27 bug. I recommend upgrading to a version of glibc that is not affected by this bug.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4376296" author="JIRAUSER1264730" created="Fri, 25 Feb 2022 06:57:28 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;In our case after applying&#160; patch (rebuilding and reinstalling same package with fix) for glibc 2.27, timeouts frequency got increased and for ubuntu 18.04 we could not be able to downgrade/upgrade glibc to different version as it is requiring other dependency to downgrade/upgrade in parallel that could disturb other running services.&#160;&lt;/p&gt;

&lt;p&gt;could we reduce the value for maxIdleThreadAge, if yes then what could be the minimum recommendation for the same?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="4375125" author="JIRAUSER1257066" created="Thu, 24 Feb 2022 18:45:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your patience. This appears to be the same bug with glibc 2.27 that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; attempts to work around. Prior to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;, the server would have stalled indefinitely and we would have seen the oplog buffer completely fill and oplog application stall. After &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;, oplog application threads will timeout due to &lt;tt&gt;maxIdleThreadAge&lt;/tt&gt;, which defaults to 30 seconds, the same duration as the stalls you observed.&lt;/p&gt;

&lt;p&gt;According to the thread that you linked, the glibc patch you applied for 2.27 doesn&apos;t fix this bug, but reduces the likelihood of it happening on Ubuntu 18.04. Have you observed that the hangs occur less frequently after applying the patch?&lt;/p&gt;

&lt;p&gt;You may be able to work around this by updating or downgrading glibc to a version that is not affected by this bug.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4372937" author="JIRAUSER1257066" created="Wed, 23 Feb 2022 21:56:01 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading the mongod.log files and FTDC. This gives us good context around the gdb samples.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4370673" author="JIRAUSER1264730" created="Wed, 23 Feb 2022 05:35:57 +0000"  >&lt;p&gt;Hi Edward,&lt;/p&gt;

&lt;p&gt;I have uploaded mongologs on support uploader for unresponsive node, Please find the details given below:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;File Name:&lt;/b&gt; mongolog20_02_2022_13_25.tar.gz&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Unresponsive time duration&lt;/b&gt; - from 2022-02-20_13-25-11 to 2022-02-20_13-25-40 UTC&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kapil&lt;/p&gt;</comment>
                            <comment id="4369169" author="JIRAUSER1257066" created="Tue, 22 Feb 2022 18:08:19 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading the gdb output. Can you please also upload an archive (tar or zip) of the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory covering the timestamps where you collected stack traces? It will aid in the investigation. In the meantime, I will take a look at the stack traces.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4367580" author="JIRAUSER1264730" created="Tue, 22 Feb 2022 05:24:56 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;Gentle Reminder!&lt;/p&gt;

&lt;p&gt;It is just a gentle reminder for any WA as this issue is badly impacting our product and we have to validate this release to move further.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;</comment>
                            <comment id="4365907" author="JIRAUSER1264730" created="Sun, 20 Feb 2022 15:55:26 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;I have uploaded gdboutput on support uploader for unresponsive node, Please find the details given below:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;File Name:&lt;/b&gt; gdboutput.tar.gz&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Unresponsive time duration:&lt;/b&gt; from 2022-02-20_13-25-11 to 2022-02-20_13-25-40 UTC&lt;/p&gt;

&lt;p&gt;Files list of gdboutput for 5 timestamps is given below:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;gdboutput:&lt;/b&gt;&lt;br/&gt;
 total 4724&lt;br/&gt;
 27031-gdb_2022-02-20_13-25-16.txt&lt;br/&gt;
 27031-gdb_2022-02-20_13-25-22.txt&lt;br/&gt;
 27031-gdb_2022-02-20_13-25-27.txt&lt;br/&gt;
 27031-gdb_2022-02-20_13-25-33.txt&lt;br/&gt;
 27031-gdb_2022-02-20_13-25-38.txt&lt;/p&gt;

&lt;p&gt;Please let me know if any other information is required.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
 Kapil&lt;/p&gt;</comment>
                            <comment id="4362439" author="JIRAUSER1257066" created="Thu, 17 Feb 2022 19:50:31 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the update. Unfortunately, I&apos;m not able to provide any workarounds since there&apos;s not enough information about the issue so far. I look forward to receiving the gdb output.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4355241" author="JIRAUSER1264730" created="Tue, 15 Feb 2022 06:09:56 +0000"  >&lt;p&gt;Hi Edwin,&lt;/p&gt;

&lt;p&gt;Thank you for the analysis.&#160;&lt;/p&gt;

&lt;p&gt;As part of 4.0.26 , MinThreadCount value is showing zero as given below:&lt;/p&gt;

&lt;p&gt;rs-app_shardAB-ipv6-5:SECONDARY&amp;gt; db.adminCommand({getParameter:1 ,replWriterMinThreadCount: 1})&lt;br/&gt;
{&lt;br/&gt;
 &lt;b&gt;&quot;replWriterMinThreadCount&quot; : 0,&lt;/b&gt;&lt;br/&gt;
 &quot;ok&quot; : 1,&lt;br/&gt;
 &quot;operationTime&quot; : Timestamp(1644901703, 7),&lt;br/&gt;
 &quot;$clusterTime&quot; : {&lt;br/&gt;
 &quot;clusterTime&quot; : Timestamp(1644901703, 7),&lt;br/&gt;
 &quot;signature&quot; : &lt;/p&gt;
{
 &quot;hash&quot; : BinData(0,&quot;B5XppE+gur52j/qqWJxPbMIk/gQ=&quot;),
 &quot;keyId&quot; : NumberLong(&quot;7042384427015471107&quot;)
 }
&lt;p&gt;}}&lt;/p&gt;

&lt;p&gt;For glibc 2.27 issue below WA we have already tried.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Bug:&#160;&lt;a href=&quot;https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1899800&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/189980,&lt;/a&gt;Fix: &lt;a href=&quot;https://launchpadlibrarian.net/502006054/mitigation.patch&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://launchpadlibrarian.net/502006054/mitigation.patch&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Have rebuilt gibc dev packages with above fix from source code and installed (libc6_2.27-3ubuntu1.4,libc-bin_2.27-3ubuntu1.4) on top of existing glibc.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;But we could not get issue resolved with above WA.&lt;/p&gt;

&lt;p&gt;Could you please recommend any WA for the issue in the meanwhile,.&lt;/p&gt;

&lt;p&gt;I would provide above gdb command output in sometime.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kapil&lt;/p&gt;</comment>
                            <comment id="4354048" author="JIRAUSER1257066" created="Mon, 14 Feb 2022 18:02:34 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing initial diagnostic data across the cluster. I suspect this might be a reoccurrence of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt;, which fixes secondary hangs caused by a bug on glibc v2.27. However, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56054&quot; title=&quot;Change minThreads value for replication writer thread pool to 0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56054&quot;&gt;&lt;del&gt;SERVER-56054&lt;/del&gt;&lt;/a&gt; was backported to 4.0.26, so seeing this issue on 4.0.27 would be unexpected. &lt;/p&gt;

&lt;p&gt;In order to help confirm the behavior that&apos;s occurring, can you provide gdb on the node that is unresponsive the next time this behavior occurs?&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gdb -p $(pidof mongod) -batch -ex &apos;thread apply all bt&apos; &amp;gt; gdb_`date +&quot;%Y-%m-%d_%H-%M-%S&quot;`.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4351948" author="JIRAUSER1264730" created="Sat, 12 Feb 2022 07:25:40 +0000"  >&lt;p&gt;Hi Edward,&lt;/p&gt;

&lt;p&gt; I have uploaded required logs on support uploader, Please find the details given below:&lt;/p&gt;

&lt;p&gt; &lt;b&gt;File Name:&lt;/b&gt; MongoQueryHighReponseLogs.tar.gz&lt;/p&gt;

&lt;p&gt; &lt;b&gt;HighQueryReponse Issue Time:&lt;/b&gt; 2022-02-11T22:51 UTC&lt;/p&gt;

&lt;p&gt; Directory structure of logs file is given below:&lt;/p&gt;

&lt;p&gt; QueryHighReponseLogs/:&lt;br/&gt;
 &lt;b&gt;Arbiter1 Arbiter2 Arbiter3 Primary rs.status Secondary1withHighReponse Secondary2 Secondary3&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Arbiter1:&lt;br/&gt;
 mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Arbiter2:&lt;br/&gt;
 mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Arbiter3:&lt;br/&gt;
 mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Primary:&lt;br/&gt;
 diagnostic.data mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Primary/diagnostic.data:&lt;br/&gt;
 metrics.2022-02-11T12-50-49Z-00000 metrics.2022-02-12T05-00-49Z-00000 metrics.interim&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary1withHighReponse:&lt;br/&gt;
 diagnostic.data mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary1withHighReponse/diagnostic.data:&lt;br/&gt;
 metrics.2022-02-11T12-35-12Z-00000 metrics.interim&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary2:&lt;br/&gt;
 diagnostic.data mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary2/diagnostic.data:&lt;br/&gt;
 metrics.2022-02-11T12-35-45Z-00000 metrics.interim&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary3:&lt;br/&gt;
 diagnostic.data mongo-27035.log&lt;/p&gt;

&lt;p&gt;QueryHighReponseLogs/Secondary3/diagnostic.data:&lt;br/&gt;
 metrics.2022-02-11T12-50-14Z-00000 metrics.interim&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Note:&lt;/b&gt; For Arbiter only mongod logs are present (diagnostics are false as it is only voting members ,does not hold actual data)&lt;/p&gt;

&lt;p&gt;Please let me know any other info is required for analysis.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;</comment>
                            <comment id="4350814" author="JIRAUSER1257066" created="Fri, 11 Feb 2022 16:57:18 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kg3634%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;kg3634@gmail.com&quot;&gt;kg3634@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your report. Would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) for each node in the replica set and upload them to this &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNTYwNTc5ODUxOTciLCJleHAiOjE2NDcxOTA0MDJ9.VxKXdYyvPLp8is01gacl4gdQ0QtXvRIYUCIrW3AfJQM&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Edwin&lt;/p&gt;</comment>
                            <comment id="4341623" author="JIRAUSER1264730" created="Tue, 8 Feb 2022 11:48:14 +0000"  >&lt;p&gt;&lt;b&gt;Additional Missing Information :&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Testing Environment:&lt;/b&gt;&lt;br/&gt;
 java client driver on application side - 3.12.9&lt;br/&gt;
 mongo version - 4.0.27&lt;br/&gt;
 Storage Engine - mmap&lt;br/&gt;
 Replica-set : 7 members (4 non-arbiter and 3 arbiter , all voting members)&lt;br/&gt;
 One of the member CMD as an example:&lt;br/&gt;
 mongod --keyFile=/mongodb.key --storageEngine mmapv1 --nojournal --noprealloc --smallfiles --ipv6 --bind_ip_all --port 27035 --dbpath=/mmapv1-tmpfs-27035 --replSet rs-app_shardAB-ipv6-7 --quiet --slowms 500 --logpath /data/db/mongo-27035.log --oplogSize 3221 --setParameter diagnosticDataCollectionEnabled=true --logappend --logRotate reopen&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Messages:&lt;/b&gt;&lt;br/&gt;
 As a sample, we got these kind of messages on mongo secondary logs given below:&lt;/p&gt;

&lt;p&gt;2022-02-02T02:55:54.392+0000 I COMMAND &lt;span class=&quot;error&quot;&gt;&amp;#91;conn554&amp;#93;&lt;/span&gt; command drasessions_1.drasessions command:&#160;&lt;b&gt;find { find&lt;/b&gt;: &#8220;drasessions&#8221;, filter: { _id: &lt;/p&gt;
{ sessionid: &#8220;ClpGx3:172.16.241.40:15124:1643368779:0080300316&#8221; }
&lt;p&gt; }, limit: 1, singleBatch: true, $db: &#8220;drasessions_1&#8221;, $clusterTime: { clusterTime: Timestamp(1643770525, 464), signature: &lt;/p&gt;
{ hash: BinData(0, A9E0739EB1E3BBA9EF776A9FCEC9342E9457D221), keyId: 7042384422720503811 }
&lt;p&gt; }, lsid: { id: UUID(&#8220;8b321501-be08-4fa8-ada5-367cc1eb555e&#8221;) }, $readPreference: { mode: &#8220;nearest&#8221; } } planSummary: IDHACK keysExamined:0 docsExamined:0 cursorExhausted:1 numYields:0 nreturned:0 reslen:239 locks:{ Global: { acquireCount: &lt;/p&gt;
{ r: 2 }
&lt;p&gt;, acquireWaitCount: { r: 1 }, timeAcquiringMicros: { r: 28911648 } }, MMAPV1Journal: { acquireCount: &lt;/p&gt;
{ r: 1 }
&lt;p&gt; }, Database: { acquireCount: &lt;/p&gt;
{ r: 1 }
&lt;p&gt; }, Collection: { acquireCount: &lt;/p&gt;
{ R: 1 }
&lt;p&gt; } } protocol:&lt;b&gt;op_msg 28911ms&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Troubleshooting performed so far:&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;We have already checked network latency/CPU/RAM/disk space on VM and blade level, so there was no issue also we tested mongo 3.6.9 in same environment and configuration but there is no issue observed.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;We enabled mongostat and also attached it as a reference and found below suspect point:&lt;/li&gt;
&lt;/ul&gt;


&lt;ol&gt;
	&lt;li&gt;For one of the secondary there are no CRUD operation for 26 sec (*0 *0 *0 *0 in mongostat observed), then suddenly high CRUD operation(*2444 110 *5781 *1816 in mongostat observed) were found on that secondary, there is no connection lost message found in mongostat for that secondary. This pattern is common whenever we are getting high response time(28 sec) on that secondary.&lt;/li&gt;
	&lt;li&gt;For Primary it is always showing *0 for insert operation for all the time but on secondary&#8217;s replication is happening for insert like given below:&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;b&gt;host insert query update delete getmore command flushes mapped vsize res faults qrw arw net_in net_out conn set repl time&lt;/b&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;2606:ae00:3001:8311:172:16:244:59&amp;#93;&lt;/span&gt;:27032 *245 36 *595 *155 0 65|0 0 6.63G 4.49G 0 0|0 0|0 24.9k 124k 144 rs-app_shardAB-ipv6-4 SEC Feb 3 17:06:46.063&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;2606:ae00:3001:8311:172:16:244:60&amp;#93;&lt;/span&gt;:27032 *&lt;b&gt;0 *0 *0 *0&lt;/b&gt; 0 63|0 0 6.61G 4.49G 0 0|0 0|0 11.3k 71.9k 184 rs-app_shardAB-ipv6-4 SEC Feb 3 17:06:46.068&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;2606:ae00:3001:8311:172:16:244:b&amp;#93;&lt;/span&gt;:27032 *227 40 *589 *155 0 64|0 0 6.75G 4.48G 0 0|0 0|0 26.6k 157k 144 rs-app_shardAB-ipv6-4 SEC Feb 3 17:06:46.075&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;2606:ae00:3001:8311:172:16:244:c&amp;#93;&lt;/span&gt;:27032 *&lt;b&gt;0&lt;/b&gt; 39 827 150 68 119|0 0 6.79G 4.49G 0 0|0 0|0 592k 1.49m 371 rs-app_shardAB-ipv6-4 &lt;b&gt;PRI&lt;/b&gt; Feb 3 17:06:45.627&lt;br/&gt;
 localhost:27032 *&lt;b&gt;0&lt;/b&gt; 41 795 162 70 120|0 0 6.79G 4.49G 0 0|0 1|0 589k 1.50m 371 rs-app_shardAB-ipv6-4 &lt;b&gt;PRI&lt;/b&gt; Feb 3 17:06:47.584&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Queries:&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;There is no CRUD on one of the secondary for 26 sec when there is no connection lost between primary and secondary.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Always 0 insert operation is showing in Primary in mongostat and even replication for insert is happening on secondary.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;This behavior is not observed in 3.6.9 where same configuration and environment was used.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Kindly reply to above queries it would really help us to process further as this is becoming a big blocker for us to use mongo in our environment.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Attachments:&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;We have attached mongostat output for one of the occurrence when we are getting high query response.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;rs.status for one of the replica-set.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1678281">SERVER-56054</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="362066" name="Screen Shot 2022-02-22 at 3.12.10 PM.png" size="156403" author="edwin.zhou@mongodb.com" created="Tue, 22 Feb 2022 20:15:03 +0000"/>
                            <attachment id="362233" name="Screen Shot 2022-02-23 at 11.39.56 AM.png" size="253563" author="edwin.zhou@mongodb.com" created="Wed, 23 Feb 2022 16:42:17 +0000"/>
                            <attachment id="359204" name="mogostat.txt" size="46284" author="kg3634@gmail.com" created="Tue, 8 Feb 2022 11:29:48 +0000"/>
                            <attachment id="359203" name="rs.status.txt" size="9327" author="kg3634@gmail.com" created="Tue, 8 Feb 2022 11:29:47 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 11 Feb 2022 16:57:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 49 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 49 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>edwin.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>kg3634@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0jh7b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i02k33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[edwin.zhou@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0j3cn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>