<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:23:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-72978] Random high query response and very high CPU on running rs combination with  4.2.20 and 4.4.18 versions</title>
                <link>https://jira.mongodb.org/browse/SERVER-72978</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Scenario&lt;/b&gt; - We run tests with same load on 2 mongo versions(4.2.20,4.4.20).&lt;br/&gt;
We found high query response(in range of 3-4 seconds)&#160; and very high cpu(100%) on primary for 4.2.20 version but no issue was found on 4.4.18&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;As per the suggestion in Server-67217, we also collected disk usage but we did not find any high disk read/write at the issue time.&lt;/li&gt;
	&lt;li&gt;In Server-67217, we did same test between (4.2.20 and 4.0.27) and still issue was found on 4.2.20 primary members only.&lt;/li&gt;
	&lt;li&gt;Kindly refer Server-67217 for more details about the issue.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;Note&lt;/b&gt;: Please provide cloud link to upload respective logs.&lt;/p&gt;</description>
                <environment>java client driver on client side - 3.12.9&lt;br/&gt;
mongo version - 4.2.20 (for 2 data members)&lt;br/&gt;
mongo version - 4.4.18 (for 2 data members)&lt;br/&gt;
Storage Engine - WT&lt;br/&gt;
Storage Type - tmpfs&lt;br/&gt;
Replica-set : 7 members (4 non-arbiter and 3 arbiter , all voting members)&lt;br/&gt;
One of the member CMD as an example:&lt;br/&gt;
mongod --keyFile=/mongodb.key --enableMajorityReadConcern false --ipv6 --bind_ip_all --port 27021 --dbpath=/data/db/wt-27021 --replSet rs-shard-2 --quiet --slowms 500 --logpath /data/db/mongo-27021.log --setParameter diagnosticDataCollectionEnabled=true --logappend --oplogSize 3221 --logRotate reopen --wiredTigerCacheSizeGB 4.40</environment>
        <key id="2235698">SERVER-72978</key>
            <summary>Random high query response and very high CPU on running rs combination with  4.2.20 and 4.4.18 versions</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="11262" iconUrl="https://jira.mongodb.org/images/icons/statuses/generic.png" description="">Investigating</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="chris.kelly@mongodb.com">Chris Kelly</assignee>
                                    <reporter username="kg3634@gmail.com">KAPIL GUPTA</reporter>
                        <labels>
                            <label>Bug</label>
                    </labels>
                <created>Wed, 18 Jan 2023 06:28:22 +0000</created>
                <updated>Fri, 17 Mar 2023 05:02:10 +0000</updated>
                                            <version>4.2.20</version>
                    <version>4.4.18</version>
                                                    <component>Performance</component>
                    <component>Querying</component>
                    <component>WiredTiger</component>
                                        <votes>25</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="5279807" author="JIRAUSER1264730" created="Fri, 17 Mar 2023 05:02:10 +0000"  >&lt;p&gt;Gentle Reminder 2!&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;</comment>
                            <comment id="5245929" author="JIRAUSER1264730" created="Fri, 3 Mar 2023 04:35:44 +0000"  >&lt;p&gt;Hi Chris,&lt;/p&gt;

&lt;p&gt;Gentle Reminder!&lt;/p&gt;

&lt;p&gt;Did you get anything on this?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;</comment>
                            <comment id="5223794" author="JIRAUSER1264730" created="Thu, 23 Feb 2023 05:41:49 +0000"  >&lt;p&gt;Hi Chris,&lt;/p&gt;

&lt;p&gt;I have uploaded the logs. Please find the logs file detail given below.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Parent file name:&lt;/b&gt; Logs.zip&lt;/p&gt;

&lt;p&gt;&lt;b&gt;child files:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;cps@vpas-B-persistence-db-9:~$ tar -tf PrimaryLogs.tar.gz&lt;br/&gt;
diagnostic.data/&lt;br/&gt;
diagnostic.data/metrics.interim&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T11-44-24Z-00000&lt;br/&gt;
journalctlPrimary.logs&lt;br/&gt;
kernalDmesgLogs&lt;br/&gt;
mongo-27040.log&#160;&lt;br/&gt;
mongo-27040.log.1&lt;br/&gt;
mongo-27040.log_verbose&lt;/p&gt;

&lt;p&gt;&lt;b&gt;NOTE: high cpu is seen multiple times(almost every 2 minutes) from 22.02.23 12:45:00 to 22.02.23 13:45:00 (mentioned in mongo-27040.log and mongo-27040.log.1) and also I enabled log verbosity from 2023-02-22T13:09:21.295+0000 (mentioned in mongo-27040.log_verbose ) to get more logs&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;cps@vpas-B-persistence-db-10:~$ tar -tf Secondary4.2Logs.tar.gz&lt;br/&gt;
diagnostic.data/&lt;br/&gt;
diagnostic.data/metrics.interim&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T13-09-06Z-00000&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T04-56-59Z-00000&lt;br/&gt;
mongo-27040.log&lt;/p&gt;


&lt;p&gt;cps@vpas-A-persistence-db-10:~$ sudo tar -tf Secondary4.4Logs.tar.gz&lt;br/&gt;
diagnostic.data/&lt;br/&gt;
diagnostic.data/metrics.interim&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T09-59-46Z-00000&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T00-13-58Z-00000&lt;br/&gt;
mongo-27040.log&lt;/p&gt;


&lt;p&gt;cps@vpas-A-persistence-db-9:~$ sudo tar -tf Secondary_2_4.4Logs.tar.gz&lt;br/&gt;
diagnostic.data/&lt;br/&gt;
diagnostic.data/metrics.interim&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T00-14-27Z-00000&lt;br/&gt;
diagnostic.data/metrics.2023-02-22T09-59-46Z-00000&lt;br/&gt;
mongo-27040.log&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Kapil&lt;/p&gt;</comment>
                            <comment id="5221839" author="JIRAUSER1265262" created="Wed, 22 Feb 2023 17:13:48 +0000"  >&lt;p&gt;Hi Kapil,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a new secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxOTA3NTgxMjk2NjMiLCJleHAiOjE2Nzk2Nzc4MjZ9.yGW0x1HU3GJuESghEhCnNlx8PeNpPNc9u4WbdCz1YIg&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; link for you.&#160;&lt;/p&gt;

&lt;p&gt;Christopher&lt;/p&gt;</comment>
                            <comment id="5220767" author="JIRAUSER1264730" created="Wed, 22 Feb 2023 13:59:34 +0000"  >&lt;p&gt;Hi Chris,&lt;/p&gt;

&lt;p&gt;Sorry for delay as we lost our last logs. we recreated the issue now but unfortunately the upload link has been expired.&lt;/p&gt;

&lt;p&gt;As per your suggestion, we checked that feasibility( have lower version secondaries)as well but&#160; here scenario is like -&lt;span class=&quot;error&quot;&gt;&amp;#91;1 primary and 1 secondary on 4.2 and 2 secondaries on 4.4&amp;#93;&lt;/span&gt;also we tested same with (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-67217&quot; title=&quot;High query response on primary (3-4 sec range) in mongo 4.2.20 version &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-67217&quot;&gt;&lt;del&gt;SERVER-67217&lt;/del&gt;&lt;/a&gt;&#160;)[ 1 primary and 1 secondary on 4.0 and 2 secondaries on&#160; 4.2 ].&lt;/p&gt;

&lt;p&gt;Each time we got high cpu issue on 4.2 primary member only.&lt;/p&gt;

&lt;p&gt;Kindly provide the new upload link to upload the logs.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kapil.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="5179939" author="JIRAUSER1265262" created="Wed, 8 Feb 2023 19:28:31 +0000"  >&lt;p&gt;We still need additional information to diagnose the problem. If this is still an issue for you, would you please supply the requested information if possible?&lt;/p&gt;

&lt;p&gt;Also, just as an FYI, it seems like you are pointing out that you are observing latency on a higher version primary node, when you have lower version secondaries. Per the &lt;a href=&quot;https://www.mongodb.com/docs/manual/release-notes/4.4-upgrade-replica-set/#upgrade-process&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upgrade steps&lt;/a&gt;, you should be upgrading these secondaries first to the higher version, and then finishing your upgrade by stepping down the primary (which should be the last one to be upgraded)&lt;/p&gt;

&lt;p&gt;Effectively, this sounds like it should mitigate your problem. If it does not, please provide further detail regarding this process (and the requested information above). Thanks!&lt;/p&gt;</comment>
                            <comment id="5124482" author="JIRAUSER1265262" created="Thu, 19 Jan 2023 16:51:41 +0000"  >&lt;p&gt;Hi Kapil,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxOTA3NTgxMjk2NjMiLCJleHAiOjE2NzY3Mzg5MzR9.VxJEah083tbG2jiUtNXWqk9M_SjBhdYpMIPuVs7XmAE&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the mongod logs&lt;/li&gt;
	&lt;li&gt;the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;Additionally, if you have a test driver that reproduces the workload you&apos;re having issues with, that would be significantly helpful in pinning down what may be occurring here.&lt;/p&gt;

&lt;p&gt;Christopher&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2066453">SERVER-67217</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 18 Jan 2023 15:33:32 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        46 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kg3634@gmail.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            46 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>chris.kelly@mongodb.com</customfieldvalue>
            <customfieldvalue>kg3634@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1r1sf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i19kfk:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;1- run 2 data members with 4.2.20 and 2 data members with 4.4.18&lt;/p&gt;

&lt;p&gt;2- put medium to high load(read/writes) on rs and observe&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[chris.kelly@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1qnxr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>