<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:52:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39598] Unwanted High Network IO in MongoDB Shard Cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-39598</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;I&apos;m having a MogoDB Shard cluster in my production environment. The mongoDB shard cluster is being resided in AWS Cloud under EC2 Instances. Following diagram illustrates big picture of the MongoDB sharded Cluster.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209261/209261_INC-mongo-db-cluster-1.jpg&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;According to the production environment, I&apos;m maintaining a cluster environment including two shards with replica set.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Shard&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Master-Node&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;Availability Zone&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Shard-01&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Replica-C&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;us-east-a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Shard-02&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Replica-A&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;us-east-a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;MongoS&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;MongoS&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;us-east-a&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;In this mongoDB which includes total around 30GB of data, but I do observe very high network in/out in between those two master nodes in the cluster and the Mongos server(Green coloured servers). This huge network traffic is being progressed with 25Mbps for continuous 10hours everyday, starting each day @5.05am UTC and it transfers around 2TB of data in total daily. This is massive cost hitting network traffic and now it&apos;s being terrible.&lt;/p&gt;

&lt;p&gt;Following network traffic monitoring dashboard gives the clear idea of spikes.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/209260/209260_2019-02-15_16-12-24.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Need to find out a solution immediately for this unwanted network traffic. Any help would be greatly appreciated.&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;</description>
                <environment></environment>
        <key id="698484">SERVER-39598</key>
            <summary>Unwanted High Network IO in MongoDB Shard Cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="madurad">Madura Dissanayake</reporter>
                        <labels>
                    </labels>
                <created>Fri, 15 Feb 2019 10:51:07 +0000</created>
                <updated>Mon, 6 May 2019 19:39:43 +0000</updated>
                            <resolved>Mon, 6 May 2019 19:39:43 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="2224747" author="daniel.hatcher" created="Thu, 25 Apr 2019 15:34:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=madurad&quot; class=&quot;user-hover&quot; rel=&quot;madurad&quot;&gt;madurad&lt;/a&gt; are you still experiencing this issue?&lt;/p&gt;</comment>
                            <comment id="2162650" author="madurad" created="Tue, 26 Feb 2019 08:17:03 +0000"  >&lt;p&gt;Hi @Daniel,&lt;/p&gt;

&lt;p&gt;Thanks for your response, as you stated 05.05 am UTC is the problem where we see and hour later is our backup time, @06.30 am UTC.&lt;/p&gt;

&lt;p&gt;Additionally I&apos;ll enable default logging options for the cluster, and will upload all the required diagnostics logs soon, to check from your side.&lt;/p&gt;

&lt;p&gt;Thanks.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2160281" author="daniel.hatcher" created="Fri, 22 Feb 2019 20:16:31 +0000"  >&lt;p&gt;Hello Madura,&lt;/p&gt;

&lt;p&gt;From the shard Primaries, I can see a very large burst of documents being returned starting around 05:05UTC every day. This lasts for about 30 minutes and then an hour later another burst of 15 minutes happens before quieting down until the next day. Unfortunately, I am not sure what is happening the rest of the day as the &lt;tt&gt;mongos&lt;/tt&gt; data would most likely be the most helpful thing there. I recommend using the built-in logging capabilities instead of redirecting the output as it would provide much more useful information in diagnosing issues.&lt;/p&gt;

&lt;p&gt;Additionally, I see that you are using two config servers for your production environment. This is &lt;b&gt;not&lt;/b&gt; recommended; you should add another config server as soon as possible.&lt;/p&gt;

&lt;p&gt;If you can collect the &lt;tt&gt;mongos&lt;/tt&gt; diagnostics covering one of these timeframes, please upload those (for all the &lt;tt&gt;mongos&lt;/tt&gt; nodes) as well as the updated shard primary diagnostics. Unfortunately, without further &lt;tt&gt;mongos&lt;/tt&gt; diagnostics, there is not much more I can check.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2158038" author="madurad" created="Thu, 21 Feb 2019 05:14:44 +0000"  >&lt;p&gt;HI @Daniel,&lt;/p&gt;

&lt;p&gt;Thanks for the feedback to my ticket. As you requested all the required logs files were uploaded to the secure portal. But in my MongoDB cluster, I start mongos server using a following command, therefore diagnostic logs are not available in the server after that.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;sudo mongos --configdb &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;replconfig01/prod-mongodb-config-01:27017,prod-mongodb-config-02:27017&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; --bind_ip &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;0.0.0.0&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&#160; &amp;amp;&amp;gt; /opt/mongodb-cluster/logs/mongodb-out.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;Please let me know, you need any further details.&lt;/p&gt;</comment>
                            <comment id="2156299" author="daniel.hatcher" created="Tue, 19 Feb 2019 21:31:06 +0000"  >&lt;p&gt;Hello Madura,&lt;/p&gt;

&lt;p&gt;Please upload the following to our &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/9f16e381-efb4-4f44-a8ff-704a487cfcf2.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Secure Upload Portal&lt;/a&gt;. Please note that only MongoDB engineers will be able to see the files uploaded there.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Shard Primaries &lt;tt&gt;mongod&lt;/tt&gt; logs&lt;/li&gt;
	&lt;li&gt;Shard Primaries &quot;diagnostic.data&quot; folders&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;mongos&lt;/tt&gt; logs&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;mongos&lt;/tt&gt; &quot;diagnostic.data&quot; folders&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Danny &lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="209260" name="2019-02-15_16-12-24.png" size="96199" author="madurad" created="Fri, 15 Feb 2019 10:50:11 +0000"/>
                            <attachment id="209261" name="INC-mongo-db-cluster-1.jpg" size="76884" author="madurad" created="Fri, 15 Feb 2019 10:49:51 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 19 Feb 2019 21:31:06 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 41 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 41 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>madurad</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunohr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hudg07:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hunar3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>