<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:31:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-75881] mongod 4.4 CPU hogs/stalls leading to client queries timedout</title>
                <link>https://jira.mongodb.org/browse/SERVER-75881</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi Team,&lt;/p&gt;

&lt;p&gt;Since migrating to MongoDB 4.4, we have been encountering a recurring issue of client queries timing out frequently. Upon monitoring the CPU usage during these occurrences, we have noticed that mongod&apos;s CPU usage is unusually high, despite there not being a significant increase in operations.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;This erratic CPU usage has been observed intermittently, and our application has a monitoring thread that checks the replicStatus and reports NO_PRIMARY during such times.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mongo- Server Version&lt;/b&gt;: 4.4.18&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mongo Client Java Driver version:&lt;/b&gt;&#160;3.12.9&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Standalone or Replica :&lt;/b&gt; 4 data bearing + 1 arbiter&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Server Option:&lt;/b&gt;&#160;&lt;/p&gt;

&lt;p&gt;root&#160;&#160;&#160;&#160;&#160;&#160; 52039&#160; 161&#160; 2.5 12649240 7229872 ?&#160;&#160;&#160; S&amp;lt;Ll Apr06 4415:45 /usr/bin/mongod --ipv6 --slowms 500 --storageEngine wiredTiger --wiredTigerCacheSizeGB 22 --enableMajorityReadConcern false --bind_ip_all --keyFile /root/.dbkey --port 27717 --dbpath=/var/data/sessions.1/h --replSet set01h --fork --pidfilepath /var/run/sessionmgr-27717.pid --oplogSize 5120 --logpath /var/log/mongodb-27717.log --logappend --quiet&lt;/p&gt;

&lt;p&gt;Storage Engine : &lt;b&gt;WT&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Application Monitors the replicaStatus periodically reporting, TYPE=UNKNOWN for PRIMARY intermittently.&lt;/b&gt;&#160;&lt;/p&gt;

&lt;p&gt;2023-04-07 23:02:30,944 &lt;span class=&quot;error&quot;&gt;&amp;#91;pool-2-thread-1&amp;#93;&lt;/span&gt; replicaset status: ReplicaSetStatus{name=set01h, cluster=ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=nd2bwa3psm16vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=795436, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;null&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:30 UTC 2023, lastUpdateTimeNanos=116414267859844}, ServerDescription{address=nd2bwa3psm15va:27717, type=&lt;b&gt;UNKNOWN&lt;/b&gt;, state=CONNECTING}, ServerDescription{address=nd2bwa3psm16va:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1152849, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:30 UTC 2023, lastUpdateTimeNanos=116414649210810}, ServerDescription{address=nd2bwa3psm15vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=943462, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:30 UTC 2023, lastUpdateTimeNanos=116414619113856}]}}&lt;br/&gt;
&lt;br/&gt;
Immediately in the next sec, the replicaSet detects PRIMARY&lt;br/&gt;
&lt;br/&gt;
2023-04-07 23:02:31,260 &lt;span class=&quot;error&quot;&gt;&amp;#91;pool-2-thread-1&amp;#93;&lt;/span&gt; replicaset status: ReplicaSetStatus{name=set01h, cluster=ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=nd2bwa3psm16vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=772997, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;null&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116414767866623}, ServerDescription{address=nd2bwa3psm15va:27717, type=&lt;b&gt;REPLICA_SET_PRIMARY&lt;/b&gt;, state=CONNECTED, ok=true, version=ServerVersion{versionList=&lt;span class=&quot;error&quot;&gt;&amp;#91;4, 4, 18&amp;#93;&lt;/span&gt;}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=16803136, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=7fffffff0000000000000080, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116414870305431}, ServerDescription{address=nd2bwa3psm16va:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1152849, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:30 UTC 2023, lastUpdateTimeNanos=116414649210810}, ServerDescription{address=nd2bwa3psm15vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=943462, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:30 UTC 2023, lastUpdateTimeNanos=116414619113856}]}}&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;But again moves to UKNOWN and later to a proper state&lt;/p&gt;

&lt;p&gt;2023-04-07 23:02:31,544 &lt;span class=&quot;error&quot;&gt;&amp;#91;pool-2-thread-1&amp;#93;&lt;/span&gt; replicaset status: ReplicaSetStatus{name=set01h, cluster=ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=nd2bwa3psm16vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=729934, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;null&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415267757469}, ServerDescription{address=nd2bwa3psm15va:27717, type=&lt;b&gt;UNKNOWN&lt;/b&gt;, state=CONNECTING}, ServerDescription{address=nd2bwa3psm16va:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1059673, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415149889939}, ServerDescription{address=nd2bwa3psm15vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=941532, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415120136591}]}}&lt;br/&gt;
&lt;br/&gt;
2023-04-07 23:02:31,764 &lt;span class=&quot;error&quot;&gt;&amp;#91;pool-3-thread-1&amp;#93;&lt;/span&gt; replicaset status: ReplicaSetStatus{name=set01h, cluster=ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=nd2bwa3psm16vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=729934, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;null&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415267757469}, ServerDescription{address=nd2bwa3psm15va:27717, type=&lt;b&gt;REPLICA_SET_PRIMARY&lt;/b&gt;, state=CONNECTED, ok=true, version=ServerVersion{versionList=&lt;span class=&quot;error&quot;&gt;&amp;#91;4, 4, 18&amp;#93;&lt;/span&gt;}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=13549634, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=7fffffff0000000000000080, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415370985974}, ServerDescription{address=nd2bwa3psm16va:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion&lt;/p&gt;
{versionList=[4, 4, 18]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1059673, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm16va:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415149889939}, ServerDescription{address=nd2bwa3psm15vb:27717, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 18]}
&lt;p&gt;, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=941532, setName=&apos;set01h&apos;, canonicalAddress=nd2bwa3psm15vb:27717, hosts=&lt;span class=&quot;error&quot;&gt;&amp;#91;nd2bwa3psm16va:27717, nd2bwa3psm15va:27717, nd2bwa3psm16vb:27717, nd2bwa3psm15vb:27717&amp;#93;&lt;/span&gt;, passives=[], arbiters=[], primary=&apos;nd2bwa3psm15va:27717&apos;, tagSet=TagSet{[]}, electionId=null, setVersion=27149688, lastWriteDate=Fri Apr 07 23:02:31 UTC 2023, lastUpdateTimeNanos=116415120136591}]}}&lt;/p&gt;


&lt;p&gt;top - 23:02:27 up 1 day, &#160;7:17, &#160;0 users, &#160;load average: 21.07, 23.62, 26.89&lt;br/&gt;
Tasks: 303 total, &#160; 1 running, 301 sleeping, &#160; 0 stopped, &#160; 1 zombie&lt;br/&gt;
%Cpu(s): 56.5 us, 15.2 sy, &#160;0.0 ni, 22.7 id, &#160;0.0 wa, &#160;2.1 hi, &#160;3.5 si, &#160;0.0 st&lt;br/&gt;
MiB Mem : 273809.9 total, 204977.1 free, &#160;30793.4 used, &#160;38039.4 buff/cache&lt;br/&gt;
MiB Swap: &#160; 4096.0 total, &#160; 4096.0 free, &#160; &#160; &#160;0.0 used. 205453.0 avail Mem&lt;/p&gt;

&lt;p&gt;&#160; &#160; PID USER &#160; &#160; &#160;PR &#160;NI &#160; &#160;VIRT &#160; &#160;RES &#160; &#160;SHR S &#160;%CPU &#160;%MEM &#160; &#160; TIME+ COMMAND&lt;br/&gt;
&#160; 52039 root &#160; &#160; &#160; 5 -15 &#160; 12.0g &#160; 7.2g &#160;36920 S 237.7 &#160; 2.7 &#160; 2977:27 mongod&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;br/&gt;
top - 23:02:30 up 1 day, &#160;7:17, &#160;0 users, &#160;load average: 21.07, 23.62, 26.89&lt;br/&gt;
Tasks: 301 total, &#160; 2 running, 299 sleeping, &#160; 0 stopped, &#160; 0 zombie&lt;br/&gt;
%Cpu(s): 87.8 us, &#160;4.8 sy, &#160;0.0 ni, &#160;5.3 id, &#160;0.0 wa, &#160;0.9 hi, &#160;1.2 si, &#160;0.0 st&lt;br/&gt;
MiB Mem : 273809.9 total, 204977.1 free, &#160;30793.4 used, &#160;38039.4 buff/cache&lt;br/&gt;
MiB Swap: &#160; 4096.0 total, &#160; 4096.0 free, &#160; &#160; &#160;0.0 used. 205453.0 avail Mem&lt;/p&gt;

&lt;p&gt;&#160; &#160; PID USER &#160; &#160; &#160;PR &#160;NI &#160; &#160;VIRT &#160; &#160;RES &#160; &#160;SHR S &#160;%CPU &#160;%MEM &#160; &#160; TIME+ COMMAND&lt;br/&gt;
&#160; &lt;b&gt;52039 root &#160; &#160; &#160; 5 -15 &#160; 12.1g &#160; 7.3g &#160;36920 S 999.9 &#160; 2.7 &#160; 2977:48 mongod&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;/p&gt;

&lt;p&gt;PID : 52039 mongostats&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160; 65 &#160;1449 &#160; &#160;969 &#160; &#160; 68 &#160; &#160; 701 &#160; 821|0 &#160;1.9% 30.2% &#160; &#160; &#160; 0 12.0G 7.19G 0|1 1|1 &#160;5.79m &#160; 14.8m &#160;741 set01h &#160;PRI Apr &#160;7 23:02:21.449&lt;br/&gt;
&#160; &#160; 71 &#160;1572 &#160; 1063 &#160; &#160; 76 &#160; &#160; 870 &#160; 959|0 &#160;1.7% 30.2% &#160; &#160; &#160; 0 12.0G 7.20G 0|0 2|0 &#160;6.45m &#160; 16.2m &#160;744 set01h &#160;PRI Apr &#160;7 23:02:22.426&lt;br/&gt;
&#160; &#160; 69 &#160;1581 &#160; 1064 &#160; &#160; 74 &#160; &#160; 841 &#160; 988|0 &#160;1.8% 30.2% &#160; &#160; &#160; 0 12.0G 7.20G 0|0 1|0 &#160;6.53m &#160; 16.2m &#160;744 set01h &#160;PRI Apr &#160;7 23:02:23.423&lt;br/&gt;
&#160; &#160; 88 &#160;1602 &#160; 1090 &#160; &#160; 77 &#160; &#160; 793 &#160; 870|0 &#160;1.8% 30.3% &#160; &#160; &#160; 0 12.0G 7.20G 0|0 1|1 &#160;6.55m &#160; 16.5m &#160;741 set01h &#160;PRI Apr &#160;7 23:02:24.431&lt;br/&gt;
&#160; &#160; 67 &#160;1507 &#160; 1026 &#160; &#160; 78 &#160; &#160; 954 &#160;1013|0 &#160;1.9% 30.1% &#160; &#160; &#160; 0 12.0G 7.20G 0|0 1|1 &#160;6.44m &#160; 15.8m &#160;741 set01h &#160;PRI Apr &#160;7 23:02:25.427&lt;br/&gt;
&#160; &#160; 59 &#160;1601 &#160; 1122 &#160; &#160; 86 &#160; &#160; 778 &#160; 817|0 &#160;1.9% 30.1% &#160; &#160; &#160; 0 12.0G 7.19G 0|1 1|0 &#160;6.48m &#160; 16.6m &#160;741 set01h &#160;PRI Apr &#160;7 23:02:26.419&lt;br/&gt;
&#160; &#160; 81 &#160;1609 &#160; 1068 &#160; &#160; 73 &#160; &#160; 779 &#160; 908|0 &#160;2.0% 30.2% &#160; &#160; &#160; 0 12.0G 7.19G 0|1 1|0 &#160;6.46m &#160; 16.3m &#160;741 set01h &#160;PRI Apr &#160;7 23:02:27.435&lt;br/&gt;
&#160; &#160; 15 &#160; 417 &#160; &#160;189 &#160; &#160; 32 &#160; &#160; 225 &#160; 364|0 &#160;1.9% 30.1% &#160; &#160; &#160; 0 11.9G 7.31G 0|0 2|8 &#160;1.44m &#160; 2.93m &#160;612 set01h &#160;PRI Apr &#160;7 23:02:30.352&lt;br/&gt;
&#160; &#160; *&lt;b&gt;0 &#160; 184 &#160; &#160; *0 &#160; &#160; &#160;9 &#160; &#160; &#160;18 &#160;2643|0 &#160;1.9% 30.2% &#160; &#160; &#160; 0 11.8G 7.32G 0|0 1|1 &#160; 318k &#160; 2.89m &#160;413 set01h &#160;PRI Apr &#160;7 23:02:30.564&lt;/b&gt;&lt;br/&gt;
&#160; &#160;140 &#160;1499 &#160; &#160;809 &#160; &#160; 94 &#160; &#160; 364 &#160;6105|0 &#160;2.0% 30.2% &#160; &#160; &#160; 0 11.6G 7.31G 0|0 1|5 &#160;5.90m &#160; 17.7m &#160;292 set01h &#160;PRI Apr &#160;7 23:02:31.422&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The response time goes to minutes&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160;total &#160; &#160; read &#160; &#160;write &#160; &#160;2023-04-07T23:02:27Z&lt;br/&gt;
session_cache_2.session &#160; &#160;321ms &#160; &#160; 86ms &#160; &#160;235ms&lt;br/&gt;
&#160; session_cache.session &#160; &#160;289ms &#160; &#160; 63ms &#160; &#160;225ms&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160;274ms &#160; &#160;274ms &#160; &#160; &#160;0ms&lt;br/&gt;
session_cache_4.session &#160; &#160;251ms &#160; &#160; 67ms &#160; &#160;184ms&lt;br/&gt;
session_cache_3.session &#160; &#160;223ms &#160; &#160; 58ms &#160; &#160;165ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160;total &#160; &#160; read &#160; &#160;write &#160; &#160;2023-04-07T23:02:29Z&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160;147ms &#160; &#160;147ms &#160; &#160; &#160;0ms&lt;br/&gt;
session_cache_2.session &#160; &#160; 96ms &#160; &#160; 27ms &#160; &#160; 68ms&lt;br/&gt;
session_cache_3.session &#160; &#160; 73ms &#160; &#160; 20ms &#160; &#160; 53ms&lt;br/&gt;
&#160; session_cache.session &#160; &#160; 71ms &#160; &#160; 17ms &#160; &#160; 53ms&lt;br/&gt;
session_cache_4.session &#160; &#160; 63ms &#160; &#160; 14ms &#160; &#160; 48ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms &#160; &#160; &#160;0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160; &#160; total &#160; &#160; &#160; &#160;read &#160; &#160; &#160;write &#160; &#160;2023-04-07T23:02:30Z&lt;br/&gt;
&lt;b&gt;session_cache_4.session &#160; &#160;187974ms &#160; &#160;126810ms &#160; &#160;61164ms&lt;/b&gt;&lt;br/&gt;
&#160; &lt;b&gt;session_cache.session &#160; &#160;159628ms &#160; &#160;116939ms &#160; &#160;42689ms&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;session_cache_2.session &#160; &#160;157818ms &#160; &#160;114013ms &#160; &#160;43804ms&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;session_cache_3.session &#160; &#160;132527ms &#160; &#160; 88974ms &#160; &#160;43552ms&lt;/b&gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160; &#160;4639ms &#160; &#160; &#160;4639ms &#160; &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160;0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160;0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160; 0ms &#160; &#160; &#160; &#160;0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160; total &#160; &#160; read &#160; &#160; write &#160; &#160;2023-04-07T23:02:31Z&lt;br/&gt;
&#160; session_cache.session &#160; &#160;1541ms &#160; &#160;374ms &#160; &#160;1166ms&lt;br/&gt;
session_cache_3.session &#160; &#160;1306ms &#160; &#160;294ms &#160; &#160;1011ms&lt;br/&gt;
session_cache_2.session &#160; &#160;1266ms &#160; &#160;247ms &#160; &#160;1018ms&lt;br/&gt;
session_cache_4.session &#160; &#160;1064ms &#160; &#160;232ms &#160; &#160; 831ms&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160; 338ms &#160; &#160;338ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160; total &#160; &#160; read &#160; &#160; write &#160; &#160;2023-04-07T23:02:32Z&lt;br/&gt;
session_cache_4.session &#160; &#160;1453ms &#160; &#160;178ms &#160; &#160;1274ms&lt;br/&gt;
&#160; session_cache.session &#160; &#160;1367ms &#160; &#160;188ms &#160; &#160;1178ms&lt;br/&gt;
session_cache_3.session &#160; &#160;1303ms &#160; &#160; 95ms &#160; &#160;1207ms&lt;br/&gt;
session_cache_2.session &#160; &#160;1210ms &#160; &#160;175ms &#160; &#160;1034ms&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160; 273ms &#160; &#160;273ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160; total &#160; &#160; read &#160; &#160; write &#160; &#160;2023-04-07T23:02:33Z&lt;br/&gt;
session_cache_2.session &#160; &#160;1407ms &#160; &#160;116ms &#160; &#160;1291ms&lt;br/&gt;
session_cache_4.session &#160; &#160;1188ms &#160; &#160;302ms &#160; &#160; 886ms&lt;br/&gt;
session_cache_3.session &#160; &#160;1147ms &#160; &#160; 79ms &#160; &#160;1068ms&lt;br/&gt;
&#160; session_cache.session &#160; &#160; 839ms &#160; &#160; 92ms &#160; &#160; 746ms&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160; 280ms &#160; &#160;280ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;ns &#160; &#160; total &#160; &#160; read &#160; &#160; write &#160; &#160;2023-04-07T23:02:34Z&lt;br/&gt;
session_cache_2.session &#160; &#160;1230ms &#160; &#160; 87ms &#160; &#160;1142ms&lt;br/&gt;
session_cache_3.session &#160; &#160;1208ms &#160; &#160;136ms &#160; &#160;1071ms&lt;br/&gt;
&#160; session_cache.session &#160; &#160;1133ms &#160; &#160; 56ms &#160; &#160;1076ms&lt;br/&gt;
session_cache_4.session &#160; &#160;1118ms &#160; &#160; 65ms &#160; &#160;1053ms&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160;local.oplog.rs &#160; &#160; 347ms &#160; &#160;347ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160; admin.system.keys &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.roles &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160; &#160;admin.system.users &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
&#160; &#160;admin.system.version &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;br/&gt;
config.image_collection &#160; &#160; &#160; 0ms &#160; &#160; &#160;0ms &#160; &#160; &#160; 0ms&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;CPU idle% goes to 0.0&lt;/b&gt;&lt;br/&gt;
Linux 4.18.0-372.26.1.el8_6.x86_64 (nd2bwa3psm15va) &#160; &#160; 04/07/2023 &#160; &#160; &#160;&lt;em&gt;x86_64&lt;/em&gt; &#160; &#160; &#160; &#160;(10 CPU)&lt;/p&gt;

&lt;p&gt;11:02:09 PM &#160; &#160; CPU &#160; &#160; %user &#160; &#160; %nice &#160; %system &#160; %iowait &#160; &#160;%steal &#160; &#160; %idle&lt;br/&gt;
11:02:10 PM &#160; &#160; all &#160; &#160; 69.90 &#160; &#160; &#160;0.00 &#160; &#160; 21.21 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;8.89&lt;br/&gt;
11:02:11 PM &#160; &#160; all &#160; &#160; 65.22 &#160; &#160; &#160;0.00 &#160; &#160; 22.38 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 12.40&lt;br/&gt;
11:02:12 PM &#160; &#160; all &#160; &#160; 54.65 &#160; &#160; &#160;0.00 &#160; &#160; 20.63 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 24.72&lt;br/&gt;
11:02:13 PM &#160; &#160; all &#160; &#160; 53.48 &#160; &#160; &#160;0.00 &#160; &#160; 21.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 25.51&lt;br/&gt;
11:02:14 PM &#160; &#160; all &#160; &#160; 50.76 &#160; &#160; &#160;0.00 &#160; &#160; 23.45 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 25.79&lt;br/&gt;
11:02:15 PM &#160; &#160; all &#160; &#160; 56.42 &#160; &#160; &#160;0.00 &#160; &#160; 21.44 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 22.14&lt;br/&gt;
11:02:16 PM &#160; &#160; all &#160; &#160; 56.44 &#160; &#160; &#160;0.00 &#160; &#160; 22.49 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 21.06&lt;br/&gt;
11:02:17 PM &#160; &#160; all &#160; &#160; 55.44 &#160; &#160; &#160;0.00 &#160; &#160; 21.15 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 23.41&lt;br/&gt;
11:02:18 PM &#160; &#160; all &#160; &#160; 52.41 &#160; &#160; &#160;0.00 &#160; &#160; 20.16 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 27.43&lt;br/&gt;
11:02:19 PM &#160; &#160; all &#160; &#160; 54.22 &#160; &#160; &#160;0.00 &#160; &#160; 21.67 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 24.11&lt;br/&gt;
11:02:20 PM &#160; &#160; all &#160; &#160; 52.24 &#160; &#160; &#160;0.00 &#160; &#160; 20.88 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 26.88&lt;br/&gt;
11:02:21 PM &#160; &#160; all &#160; &#160; 61.88 &#160; &#160; &#160;0.00 &#160; &#160; 21.44 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 16.68&lt;br/&gt;
11:02:22 PM &#160; &#160; all &#160; &#160; 62.61 &#160; &#160; &#160;0.00 &#160; &#160; 21.78 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 15.60&lt;br/&gt;
11:02:23 PM &#160; &#160; all &#160; &#160; 50.36 &#160; &#160; &#160;0.00 &#160; &#160; 19.16 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 30.48&lt;br/&gt;
11:02:24 PM &#160; &#160; all &#160; &#160; 54.06 &#160; &#160; &#160;0.00 &#160; &#160; 22.72 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 23.23&lt;br/&gt;
11:02:25 PM &#160; &#160; all &#160; &#160; 55.25 &#160; &#160; &#160;0.00 &#160; &#160; 20.39 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 24.36&lt;br/&gt;
11:02:26 PM &#160; &#160; all &#160; &#160; 50.36 &#160; &#160; &#160;0.00 &#160; &#160; 18.04 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 31.60&lt;br/&gt;
11:02:27 PM &#160; &#160; all &#160; &#160; 52.86 &#160; &#160; &#160;0.00 &#160; &#160; 19.29 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 27.86&lt;br/&gt;
11:02:28 PM &#160; &#160; all &#160; &#160; 67.58 &#160; &#160; &#160;0.00 &#160; &#160; 15.15 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 17.27&lt;br/&gt;
&lt;b&gt;11:02:29 PM &#160; &#160; all &#160; &#160; 97.20 &#160; &#160; &#160;0.00 &#160; &#160; &#160;2.80 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00&lt;/b&gt;&lt;br/&gt;
11:02:30 PM &#160; &#160; all &#160; &#160; 88.59 &#160; &#160; &#160;0.00 &#160; &#160; &#160;9.11 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;2.30&lt;br/&gt;
11:02:31 PM &#160; &#160; all &#160; &#160; 59.90 &#160; &#160; &#160;0.00 &#160; &#160; 26.40 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 13.71&lt;br/&gt;
11:02:32 PM &#160; &#160; all &#160; &#160; 63.96 &#160; &#160; &#160;0.00 &#160; &#160; 23.49 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 12.55&lt;br/&gt;
11:02:33 PM &#160; &#160; all &#160; &#160; 70.56 &#160; &#160; &#160;0.00 &#160; &#160; 21.88 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;7.56&lt;br/&gt;
11:02:34 PM &#160; &#160; all &#160; &#160; 65.70 &#160; &#160; &#160;0.00 &#160; &#160; 23.00 &#160; &#160; &#160;0.00 &#160; &#160; &#160;0.00 &#160; &#160; 11.30&lt;/p&gt;


&lt;p&gt;Linux 4.18.0-372.26.1.el8_6.x86_64 (nd2bwa3psm15va) &#160; &#160; 04/07/2023 &#160; &#160; &#160;&lt;em&gt;x86_64&lt;/em&gt; &#160; &#160; &#160; &#160;(10 CPU)&lt;/p&gt;

&lt;p&gt;11:02:09 PM &#160; runq-sz &#160;plist-sz &#160; ldavg-1 &#160; ldavg-5 &#160;ldavg-15 &#160; blocked&lt;br/&gt;
11:02:10 PM &#160; &#160; &#160; &#160;28 &#160; &#160; &#160;2985 &#160; &#160; 22.28 &#160; &#160; 23.99 &#160; &#160; 27.08 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:11 PM &#160; &#160; &#160; &#160;17 &#160; &#160; &#160;2994 &#160; &#160; 22.28 &#160; &#160; 23.99 &#160; &#160; 27.08 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:12 PM &#160; &#160; &#160; &#160;24 &#160; &#160; &#160;3025 &#160; &#160; 22.28 &#160; &#160; 23.99 &#160; &#160; 27.08 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:13 PM &#160; &#160; &#160; &#160;10 &#160; &#160; &#160;3024 &#160; &#160; 22.58 &#160; &#160; 24.02 &#160; &#160; 27.07 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:14 PM &#160; &#160; &#160; &#160; 6 &#160; &#160; &#160;2956 &#160; &#160; 22.58 &#160; &#160; 24.02 &#160; &#160; 27.07 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:15 PM &#160; &#160; &#160; &#160;25 &#160; &#160; &#160;2959 &#160; &#160; 22.58 &#160; &#160; 24.02 &#160; &#160; 27.07 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:16 PM &#160; &#160; &#160; &#160;37 &#160; &#160; &#160;2960 &#160; &#160; 22.58 &#160; &#160; 24.02 &#160; &#160; 27.07 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:17 PM &#160; &#160; &#160; &#160; 7 &#160; &#160; &#160;2964 &#160; &#160; 22.58 &#160; &#160; 24.02 &#160; &#160; 27.07 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:18 PM &#160; &#160; &#160; &#160;10 &#160; &#160; &#160;2968 &#160; &#160; 21.89 &#160; &#160; 23.86 &#160; &#160; 27.00 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:19 PM &#160; &#160; &#160; &#160;21 &#160; &#160; &#160;2968 &#160; &#160; 21.89 &#160; &#160; 23.86 &#160; &#160; 27.00 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:20 PM &#160; &#160; &#160; &#160;25 &#160; &#160; &#160;3001 &#160; &#160; 21.89 &#160; &#160; 23.86 &#160; &#160; 27.00 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:21 PM &#160; &#160; &#160; &#160;35 &#160; &#160; &#160;2999 &#160; &#160; 21.89 &#160; &#160; 23.86 &#160; &#160; 27.00 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:22 PM &#160; &#160; &#160; &#160;20 &#160; &#160; &#160;3037 &#160; &#160; 21.89 &#160; &#160; 23.86 &#160; &#160; 27.00 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:23 PM &#160; &#160; &#160; &#160;23 &#160; &#160; &#160;3037 &#160; &#160; 21.42 &#160; &#160; 23.73 &#160; &#160; 26.94 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:24 PM &#160; &#160; &#160; &#160;29 &#160; &#160; &#160;2983 &#160; &#160; 21.42 &#160; &#160; 23.73 &#160; &#160; 26.94 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:25 PM &#160; &#160; &#160; &#160;11 &#160; &#160; &#160;2981 &#160; &#160; 21.42 &#160; &#160; 23.73 &#160; &#160; 26.94 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:26 PM &#160; &#160; &#160; &#160;18 &#160; &#160; &#160;2971 &#160; &#160; 21.42 &#160; &#160; 23.73 &#160; &#160; 26.94 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
11:02:27 PM &#160; &#160; &#160; &#160;13 &#160; &#160; &#160;2972 &#160; &#160; 21.42 &#160; &#160; 23.73 &#160; &#160; 26.94 &#160; &#160; &#160; &#160; 0&lt;br/&gt;
&lt;b&gt;11:02:28 PM &#160; &#160; &#160; 300 &#160; &#160; &#160;2985 &#160; &#160; 21.07 &#160; &#160; 23.62 &#160; &#160; 26.89 &#160; &#160; &#160; &#160; 0&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;11:02:29 PM &#160; &#160; &#160; 741 &#160; &#160; &#160;3160 &#160; &#160; 21.07 &#160; &#160; 23.62 &#160; &#160; 26.89 &#160; &#160; &#160; &#160; 0&lt;/b&gt;&lt;br/&gt;
&lt;b&gt;11:02:30 PM &#160; &#160; &#160; 115 &#160; &#160; &#160;2612 &#160; &#160; 21.07 &#160; &#160; 23.62 &#160; &#160; 26.89 &#160; &#160; &#160; &#160; 0&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;SM16va: SECONDARY member mongostat&lt;/ins&gt;&lt;/p&gt;

&lt;p&gt;insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*69 &#160; &#160;18 &#160;*1018 &#160; &#160;*83 &#160; &#160; &#160; 0 &#160; 120|0 &#160;0.9% 44.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;25.2k &#160; &#160;170k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:15.245&lt;br/&gt;
&#160; &#160;*67 &#160; &#160;23 &#160; *935 &#160; &#160;*79 &#160; &#160; &#160; 0 &#160; 134|0 &#160;1.0% 44.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;29.4k &#160; &#160;167k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:16.231&lt;br/&gt;
&#160; &#160;*71 &#160; &#160;24 &#160; *984 &#160; &#160;*67 &#160; &#160; &#160; 0 &#160; 132|0 &#160;1.0% 44.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;29.5k &#160; &#160;229k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:17.227&lt;br/&gt;
&#160; &#160;*67 &#160; &#160;27 &#160;*1061 &#160; &#160;*72 &#160; &#160; &#160; 0 &#160; 136|0 &#160;1.0% 44.0% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;31.2k &#160; &#160;263k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:18.229&lt;br/&gt;
&#160; &#160;*76 &#160; &#160;24 &#160;*1051 &#160; &#160;*65 &#160; &#160; &#160; 0 &#160; 136|0 &#160;1.0% 44.0% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;31.0k &#160; &#160;170k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:19.226&lt;br/&gt;
&#160; &#160;*59 &#160; &#160;28 &#160;*1028 &#160; &#160;*65 &#160; &#160; &#160; 0 &#160; 128|0 &#160;1.0% 44.0% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;29.7k &#160; &#160;166k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:20.225&lt;br/&gt;
&#160; &#160;*67 &#160; &#160;27 &#160; *934 &#160; &#160;*65 &#160; &#160; &#160; 0 &#160; 123|0 &#160;1.1% 43.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;28.7k &#160; &#160;157k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:21.232&lt;br/&gt;
&#160; &#160;*64 &#160; &#160;28 &#160;*1079 &#160; &#160;*72 &#160; &#160; &#160; 0 &#160; 125|0 &#160;1.1% 43.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 2|0 &#160;29.2k &#160; &#160;166k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:22.226&lt;br/&gt;
&#160; &#160;*70 &#160; &#160;22 &#160; *900 &#160; &#160;*60 &#160; &#160; &#160; 0 &#160; 118|0 &#160;1.1% 43.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;26.1k &#160; &#160;158k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:23.249&lt;br/&gt;
&#160; &#160;*83 &#160; &#160;29 &#160;*1140 &#160; &#160;*69 &#160; &#160; &#160; 0 &#160; 140|0 &#160;1.1% 43.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;32.8k &#160; &#160;180k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:24.235&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*80 &#160; &#160;35 &#160;*1039 &#160; &#160;*74 &#160; &#160; &#160; 0 &#160; 121|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;30.9k &#160; &#160;169k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:25.240&lt;br/&gt;
&#160; &#160;*50 &#160; &#160;28 &#160; *993 &#160; &#160;*78 &#160; &#160; &#160; 0 &#160; 127|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;29.5k &#160; &#160;163k &#160;153 set01h &#160;SEC Apr &#160;7 23:02:26.227&lt;br/&gt;
&#160; &#160;*85 &#160; &#160;33 &#160;*1113 &#160; &#160;*81 &#160; &#160; &#160; 0 &#160; 129|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;31.6k &#160; &#160;162k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:27.251&lt;br/&gt;
&#160; &#160;*66 &#160; &#160;16 &#160; *653 &#160; &#160;*74 &#160; &#160; &#160; 0 &#160; 129|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;26.3k &#160; &#160;212k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:28.223&lt;br/&gt;
&#160; &#160; *&lt;b&gt;0 &#160; &#160; 5 &#160; &#160; *0 &#160; &#160; *0 &#160; &#160; &#160; 0 &#160; 149|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;27.9k &#160; &#160;310k &#160;156 set01h &#160;SEC Apr &#160;7 23:02:29.229&lt;/b&gt;&lt;br/&gt;
&#160; &#160; *&lt;b&gt;0 &#160; 261 &#160; &#160; *0 &#160; &#160; *0 &#160; &#160; &#160; 0 &#160; 126|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160; 123k &#160; 1.23m&lt;/b&gt; &#160;159 set01h &#160;SEC Apr &#160;7 23:02:30.227&lt;br/&gt;
&#160; &#160;*64 &#160; 947 &#160; *564 &#160; &#160;*96 &#160; &#160; &#160; 0 &#160; 175|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160; 400k &#160; 3.81m &#160;171 set01h &#160;SEC Apr &#160;7 23:02:31.238&lt;br/&gt;
&#160; &#160;*87 &#160; 251 &#160; *933 &#160; &#160;*82 &#160; &#160; &#160; 0 &#160; 115|0 &#160;1.1% 43.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160; 116k &#160; 1.13m &#160;170 set01h &#160;SEC Apr &#160;7 23:02:32.231&lt;br/&gt;
&#160; *124 &#160; &#160;41 &#160;*1562 &#160; *155 &#160; &#160; &#160; 0 &#160; 128|0 &#160;1.2% 43.6% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;34.0k &#160; &#160;198k &#160;170 set01h &#160;SEC Apr &#160;7 23:02:33.227&lt;br/&gt;
&#160; *119 &#160; &#160;33 &#160;*1553 &#160; &#160;*71 &#160; &#160; &#160; 0 &#160; 124|0 &#160;1.2% 43.6% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;31.2k &#160; &#160;179k &#160;170 set01h &#160;SEC Apr &#160;7 23:02:34.236&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;


&lt;p&gt;&lt;ins&gt;15vb:SECONDARY member mongostat&lt;/ins&gt;&lt;/p&gt;

&lt;p&gt;insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*61 &#160; &#160;29 &#160; *952 &#160; &#160;*77 &#160; &#160; 434 &#160; 532|0 &#160;0.9% 42.7% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 733k &#160; 4.49m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:15.917&lt;br/&gt;
&#160; &#160;*74 &#160; &#160;20 &#160; *978 &#160; &#160;*63 &#160; &#160; 369 &#160; 413|0 &#160;0.9% 42.7% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 540k &#160; 7.30m &#160;161 set01h &#160;SEC Apr &#160;7 23:02:16.909&lt;br/&gt;
&#160; &#160;*64 &#160; &#160;24 &#160;*1010 &#160; &#160;*77 &#160; &#160; 342 &#160; 465|0 &#160;0.9% 42.8% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 582k &#160; 4.76m &#160;161 set01h &#160;SEC Apr &#160;7 23:02:17.925&lt;br/&gt;
&#160; &#160;*76 &#160; &#160;18 &#160;*1094 &#160; &#160;*69 &#160; &#160; 389 &#160; 483|0 &#160;1.0% 42.7% &#160; &#160; &#160; 0 13.1G 9.96G 0|1 1|0 &#160; 640k &#160; 5.01m &#160;161 set01h &#160;SEC Apr &#160;7 23:02:18.916&lt;br/&gt;
&#160; &#160;*62 &#160; &#160;24 &#160; *962 &#160; &#160;*60 &#160; &#160; 247 &#160; 351|0 &#160;1.0% 42.5% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 426k &#160; 4.30m &#160;161 set01h &#160;SEC Apr &#160;7 23:02:19.939&lt;br/&gt;
&#160; &#160;*67 &#160; &#160;26 &#160;*1046 &#160; &#160;*61 &#160; &#160; 260 &#160; 356|0 &#160;1.0% 42.5% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 429k &#160; 4.66m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:20.925&lt;br/&gt;
&#160; &#160;*63 &#160; &#160;22 &#160;*1004 &#160; &#160;*72 &#160; &#160; 317 &#160; 423|0 &#160;1.0% 42.5% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 543k &#160; 4.90m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:21.912&lt;br/&gt;
&#160; &#160;*73 &#160; &#160;22 &#160; *963 &#160; &#160;*68 &#160; &#160; 286 &#160; 374|0 &#160;0.9% 42.5% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 470k &#160; 4.37m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:22.924&lt;br/&gt;
&#160; &#160;*75 &#160; &#160;33 &#160;*1097 &#160; &#160;*67 &#160; &#160; 221 &#160; 308|0 &#160;1.0% 42.5% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 358k &#160; 4.81m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:23.926&lt;br/&gt;
&#160; &#160;*86 &#160; &#160;31 &#160;*1079 &#160; &#160;*63 &#160; &#160; 361 &#160; 487|0 &#160;1.0% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|2 1|0 &#160; 640k &#160; 5.02m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:24.912&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*48 &#160; &#160;19 &#160; *948 &#160; &#160;*90 &#160; &#160; 245 &#160; 328|0 &#160;1.1% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 394k &#160; 4.18m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:25.934&lt;br/&gt;
&#160; &#160;*78 &#160; &#160;41 &#160;*1095 &#160; &#160;*72 &#160; &#160; 255 &#160; 328|0 &#160;1.1% 42.7% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 397k &#160; 7.23m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:26.913&lt;br/&gt;
&#160; &#160;*80 &#160; &#160;32 &#160;*1015 &#160; &#160;*82 &#160; &#160; 442 &#160; 521|0 &#160;1.2% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 712k &#160; 4.91m &#160;161 set01h &#160;SEC Apr &#160;7 23:02:27.924&lt;br/&gt;
&#160; &#160;*10 &#160; &#160; 4 &#160; *172 &#160; &#160;*13 &#160; &#160; &#160;91 &#160; 217|0 &#160;1.1% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 162k &#160; 1.06m &#160;158 set01h &#160;SEC Apr &#160;7 23:02:28.907&lt;br/&gt;
&#160; &#160; *0 &#160; &#160;48 &#160; &#160; *0 &#160; &#160; *0 &#160; &#160; &#160; 0 &#160; 201|0 &#160;1.1% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160;53.3k &#160; &#160;408k &#160;150 set01h &#160;SEC Apr &#160;7 23:02:29.910&lt;br/&gt;
&#160; &#160;*37 &#160; 895 &#160; *333 &#160; &#160;*86 &#160; &#160; &#160;49 &#160; 438|0 &#160;1.1% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 507k &#160; 5.37m &#160;166 set01h &#160;SEC Apr &#160;7 23:02:30.914&lt;br/&gt;
&#160; &#160;*98 &#160; 464 &#160; *750 &#160; &#160;*76 &#160; &#160; 253 &#160; 359|0 &#160;1.2% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 609k &#160; 5.84m &#160;167 set01h &#160;SEC Apr &#160;7 23:02:31.908&lt;br/&gt;
&#160; *122 &#160; &#160;36 &#160;*1411 &#160; *149 &#160; &#160; 346 &#160; 453|0 &#160;1.2% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|0 1|0 &#160; 604k &#160; 6.59m &#160;167 set01h &#160;SEC Apr &#160;7 23:02:32.912&lt;br/&gt;
&#160; *123 &#160; &#160;44 &#160;*1739 &#160; &#160;*80 &#160; &#160; 311 &#160; 411|0 &#160;1.2% 42.6% &#160; &#160; &#160; 0 13.1G 9.96G 0|1 1|0 &#160; 534k &#160; 7.54m &#160;167 set01h &#160;SEC Apr &#160;7 23:02:33.917&lt;br/&gt;
&#160; &#160;*84 &#160; &#160;45 &#160;*1324 &#160; &#160;*72 &#160; &#160; 325 &#160; 444|0 &#160;1.2% 42.4% &#160; &#160; &#160; 0 13.1G 9.96G 1|0 1|0 &#160; 573k &#160; 5.95m &#160;168 set01h &#160;SEC Apr &#160;7 23:02:34.911&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;


&lt;p&gt;&lt;ins&gt;16vb:SECONDARY member mongostat&lt;/ins&gt;&lt;/p&gt;

&lt;p&gt;nsert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*75 &#160; &#160;18 &#160;*1046 &#160; &#160;*66 &#160; &#160; &#160; 0 &#160; 135|0 &#160;0.9% 43.4% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;28.8k &#160; &#160;160k &#160;166 set01h &#160;SEC Apr &#160;7 23:02:18.807&lt;br/&gt;
&#160; &#160;*65 &#160; &#160;31 &#160;*1004 &#160; &#160;*66 &#160; &#160; &#160; 0 &#160; 131|0 &#160;0.9% 43.4% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;31.6k &#160; &#160;177k &#160;166 set01h &#160;SEC Apr &#160;7 23:02:19.784&lt;br/&gt;
&#160; &#160;*67 &#160; &#160;37 &#160;*1038 &#160; &#160;*56 &#160; &#160; &#160; 0 &#160; 128|0 &#160;0.7% 42.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;33.1k &#160; &#160;172k &#160;163 set01h &#160;SEC Apr &#160;7 23:02:20.785&lt;br/&gt;
&#160; &#160;*59 &#160; &#160;28 &#160; *973 &#160; &#160;*69 &#160; &#160; &#160; 0 &#160; 122|0 &#160;0.8% 42.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;29.1k &#160; &#160;157k &#160;163 set01h &#160;SEC Apr &#160;7 23:02:21.794&lt;br/&gt;
&#160; &#160;*76 &#160; &#160;30 &#160; *991 &#160; &#160;*73 &#160; &#160; &#160; 0 &#160; 127|0 &#160;0.9% 42.8% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;30.0k &#160; &#160;166k &#160;163 set01h &#160;SEC Apr &#160;7 23:02:22.792&lt;br/&gt;
&#160; &#160;*70 &#160; &#160;26 &#160;*1110 &#160; &#160;*70 &#160; &#160; &#160; 0 &#160; 124|0 &#160;0.9% 42.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;28.5k &#160; &#160;156k &#160;163 set01h &#160;SEC Apr &#160;7 23:02:23.783&lt;br/&gt;
&#160; &#160;*85 &#160; &#160;22 &#160;*1031 &#160; &#160;*63 &#160; &#160; &#160; 0 &#160; 122|0 &#160;1.0% 42.9% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 2|0 &#160;26.8k &#160; &#160;175k &#160;163 set01h &#160;SEC Apr &#160;7 23:02:24.794&lt;br/&gt;
&#160; &#160;*58 &#160; &#160;21 &#160; *958 &#160; &#160;*87 &#160; &#160; &#160; 0 &#160; 132|0 &#160;1.1% 43.0% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;29.3k &#160; &#160;209k &#160;166 set01h &#160;SEC Apr &#160;7 23:02:25.787&lt;br/&gt;
&#160; &#160;*66 &#160; &#160;31 &#160;*1079 &#160; &#160;*68 &#160; &#160; &#160; 0 &#160; 134|0 &#160;1.1% 43.0% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;32.4k &#160; &#160;569k &#160;164 set01h &#160;SEC Apr &#160;7 23:02:26.796&lt;br/&gt;
&#160; &#160;*85 &#160; &#160;44 &#160;*1058 &#160; &#160;*81 &#160; &#160; &#160; 0 &#160; 152|0 &#160;1.2% 43.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;40.0k &#160; &#160;235k &#160;166 set01h &#160;SEC Apr &#160;7 23:02:27.785&lt;br/&gt;
insert query update delete getmore command dirty &#160;used flushes vsize &#160; res qrw arw net_in net_out conn &#160; &#160;set repl &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;time&lt;br/&gt;
&#160; &#160;*19 &#160; &#160; 5 &#160; *215 &#160; &#160;*26 &#160; &#160; &#160; 0 &#160; 126|0 &#160;1.2% 43.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;21.6k &#160; &#160;162k &#160;166 set01h &#160;SEC Apr &#160;7 23:02:28.784&lt;br/&gt;
&#160; &#160; *0 &#160; &#160;31 &#160; &#160; *0 &#160; &#160; *0 &#160; &#160; &#160; 0 &#160; 125|0 &#160;1.2% 43.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 2|0 &#160;32.7k &#160; &#160;274k &#160;161 set01h &#160;SEC Apr &#160;7 23:02:29.788&lt;br/&gt;
&#160; &#160;*13 &#160; 808 &#160; *214 &#160; &#160;*76 &#160; &#160; &#160; 0 &#160; 148|0 &#160;1.2% 43.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160; 341k &#160; 3.32m &#160;170 set01h &#160;SEC Apr &#160;7 23:02:30.788&lt;br/&gt;
&#160; *108 &#160; 592 &#160; *792 &#160; &#160;*72 &#160; &#160; &#160; 0 &#160; 115|0 &#160;1.3% 43.1% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160; 251k &#160; 2.52m &#160;172 set01h &#160;SEC Apr &#160;7 23:02:31.787&lt;br/&gt;
&#160; *124 &#160; &#160;42 &#160;*1368 &#160; *154 &#160; &#160; &#160; 0 &#160; 128|0 &#160;1.4% 43.2% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;34.7k &#160; &#160;430k &#160;172 set01h &#160;SEC Apr &#160;7 23:02:32.784&lt;br/&gt;
&#160; *130 &#160; &#160;47 &#160;*1759 &#160; &#160;*80 &#160; &#160; &#160; 0 &#160; 119|0 &#160;1.5% 43.3% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;34.7k &#160; &#160;169k &#160;172 set01h &#160;SEC Apr &#160;7 23:02:33.789&lt;br/&gt;
&#160; &#160;*74 &#160; &#160;48 &#160;*1345 &#160; &#160;*74 &#160; &#160; &#160; 0 &#160; 127|0 &#160;1.6% 43.3% &#160; &#160; &#160; 0 13.0G 10.7G 0|1 1|0 &#160;35.9k &#160; &#160;176k &#160;172 set01h &#160;SEC Apr &#160;7 23:02:34.787&lt;br/&gt;
&#160; &#160;*80 &#160; &#160;43 &#160;*1274 &#160; &#160;*80 &#160; &#160; &#160; 0 &#160; 130|0 &#160;1.6% 43.4% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;35.9k &#160; &#160;210k &#160;175 set01h &#160;SEC Apr &#160;7 23:02:35.786&lt;br/&gt;
&#160; &#160;*92 &#160; &#160;29 &#160;*1167 &#160; &#160;*85 &#160; &#160; &#160; 0 &#160; 129|0 &#160;1.7% 43.4% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;30.5k &#160; &#160;209k &#160;172 set01h &#160;SEC Apr &#160;7 23:02:36.786&lt;br/&gt;
&#160; &#160;*75 &#160; &#160;26 &#160; *930 &#160; &#160;*58 &#160; &#160; &#160; 0 &#160; 150|0 &#160;1.7% 43.4% &#160; &#160; &#160; 0 13.0G 10.7G 0|0 1|0 &#160;33.9k &#160; &#160;227k &#160;175 set01h &#160;SEC Apr &#160;7 23:02:37.783&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Since the PRIMARY seems unavailable, infact NO primaries are seen for 1-2 sec, the application(client) queries are getting timedout resulting in bad customer experience.&lt;/p&gt;

&lt;p&gt;We tried to reduce the load(query) to the mongo but it looks the same. We use /etc/hosts for name resolution and do not use DNS at all.&#160;&lt;/p&gt;

&lt;p&gt;We need to know why time to time, mongod hogs.&#160;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Please provide a secure portal, I can share the diagnostics.data and complete mongodb log during this period.&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2310171">SERVER-75881</key>
            <summary>mongod 4.4 CPU hogs/stalls leading to client queries timedout</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="yuan.fang@mongodb.com">Yuan Fang</assignee>
                                    <reporter username="veramasu@hcl.com">venkataramans rama</reporter>
                        <labels>
                    </labels>
                <created>Sat, 8 Apr 2023 20:32:12 +0000</created>
                <updated>Wed, 31 May 2023 15:01:02 +0000</updated>
                            <resolved>Wed, 31 May 2023 15:01:01 +0000</resolved>
                                                                                        <votes>24</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5464279" author="JIRAUSER1270794" created="Wed, 31 May 2023 15:01:02 +0000"  >&lt;p&gt;Closing this ticket as further investigation has been moved to another thread.&lt;/p&gt;</comment>
                            <comment id="5336721" author="JIRAUSER1258850" created="Tue, 11 Apr 2023 00:06:04 +0000"  >&lt;p&gt;Thanks a lot Yuan for the analysis. Yes the date was April 7. Sorry for the typo.&lt;/p&gt;

&lt;p&gt;Yes, we also see primary insert/update reduced during this time but&#160; we believe its due to Mongo high CPU and not reachable to clients.&#160; Also with the same setup if we run with mongo 4.2/4.0 we are not observing this issue. So we believe its something unique to mongo 4.4 and hence we feel a regression issue.&#160;&lt;/p&gt;

&lt;p&gt;Also this is not just happening in one setup but multiple setups. So we don&apos;t feel its related to network issue.&#160; Also mongod primary reports IDHACK queries (optimized queries) also as Slow query because, its doing some heavy operations (which we don&apos;t know what) which is causing the HIGH CPU and Slow query response for even IDHACK queries.&#160;&lt;/p&gt;

&lt;p&gt;Please let me know if you need any additional details to find the cause of this problem.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="5336528" author="JIRAUSER1270794" created="Mon, 10 Apr 2023 21:58:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing the diagnostic data,&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Please refer to the timeline April 9th from 23:02:10 to 23:02:38&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I noticed that in your comment you referred to the timeline April 9th, but I believe you meant to refer to April 7th instead, since the FTDC data does not cover April 9th.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/442102/442102_Screenshot+2023-04-10+at+5.52.06+PM.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Based on my analysis of the FTDC data from the primary node, I observed some slow queries and increased latencies around point B (2023-04-07T23:02:29.747Z), which appear to be caused by network issues. This is supported by the decrease in &lt;tt&gt;ss network bytesIn&lt;/tt&gt; and &lt;tt&gt;ss network bytesOut&lt;/tt&gt; between points A and B. The increase in CPU usage is likely due to the server attempting to handle these slow queries. Additionally, while the status of the primary node&apos;s self state remained healthy, it was seen as down from secondary node 4 during a period after point B, which I suspect may also be related to network issues.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Since migrating to MongoDB 4.4, we have been encountering a recurring issue of client queries timing out frequently.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I&apos;m unsure if you meant to suggest a regression after upgrading. If so, could you please clarify and provide more information?&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Yuan&lt;/p&gt;</comment>
                            <comment id="5335660" author="JIRAUSER1258850" created="Mon, 10 Apr 2023 17:00:30 +0000"  >&lt;p&gt;Thanks Team for the portal link. I have uploaded all the 4 members mongod.log and diagnostics.data&lt;/p&gt;

&lt;p&gt;Please refer to the timeline April 9th from 23:02:10 to 23:02:38.&lt;/p&gt;</comment>
                            <comment id="5334941" author="JIRAUSER1270794" created="Mon, 10 Apr 2023 13:32:44 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=veramasu%40hcl.com&quot; class=&quot;user-hover&quot; rel=&quot;veramasu@hcl.com&quot;&gt;veramasu@hcl.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIyMDI3MjQ2MjgxOTEiLCJleHAiOjE2ODM3MjU0NjR9.8vHAfRXliTcisR40LIln5sojerACfId2RMGXQE1R-tU&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the mongod logs&lt;/li&gt;
	&lt;li&gt;the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Regards,&lt;br/&gt;
Yuan&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="445291" name="4.0_Primary_Startup_mongodb-27717.log" size="174866" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 18:05:18 +0000"/>
                            <attachment id="445290" name="4.0_parameters" size="12537" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 18:05:18 +0000"/>
                            <attachment id="445294" name="4.2_parameters" size="16040" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 18:10:45 +0000"/>
                            <attachment id="442102" name="Screenshot 2023-04-10 at 5.52.06 PM.png" size="451720" author="yuan.fang@mongodb.com" created="Mon, 10 Apr 2023 21:53:14 +0000"/>
                            <attachment id="442321" name="Screenshot 2023-04-11 at 4.31.44 PM.png" size="196843" author="yuan.fang@mongodb.com" created="Tue, 11 Apr 2023 20:32:51 +0000"/>
                            <attachment id="445356" name="clientlog.log" size="415504" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 23:55:10 +0000"/>
                            <attachment id="445491" name="clientsettings.jpg" size="236413" author="matt.holford@mongodb.com" created="Fri, 21 Apr 2023 15:32:59 +0000"/>
                            <attachment id="445301" name="mongo_4.0_diag.tar.gz" size="110283914" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 18:43:41 +0000"/>
                            <attachment id="445277" name="mongodb-27717.log" size="759037" author="veramasu@hcl.com" created="Thu, 20 Apr 2023 17:26:32 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001ss4GZQAY]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 10 Apr 2023 13:32:44 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        36 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>yuan.fang@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            36 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>veramasu@hcl.com</customfieldvalue>
            <customfieldvalue>yuan.fang@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i23xqf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i11xc4:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[yuan.fang@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i23jvr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>