<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:08:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-45265] Incosistant sharding overhead</title>
                <link>https://jira.mongodb.org/browse/SERVER-45265</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I ran a few experiments to compare sharding vs non-sharded mongo instances. I have limited the mongod memory to 256MB in both scenarios and disabled compression. I use a synthetic dataset of 80-byte documents and change the number of documents of the collections and collect the average runtime to retrieve a random document by id. In the sharded environment I have ranged sharding on the id and the data is equally distributed among the 3 servers (no replication). Here is the result that I got. (80-2m means 2,000,000 documents of 80 bytes)&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;241046_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/241046/241046_image-2019-12-20-01-14-53-779.png&quot; title=&quot;image-2019-12-20-01-14-53-779.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;241046&quot; file-preview-title=&quot;image-2019-12-20-01-14-53-779.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/241046/_thumb_241046.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;As expected there is an overhead associated with sharding. However, my question is that shouldn&apos;t this overhead be a constant? why does the difference between the runtime of the sharded and the non-sharded instance is getting more with more documents? As far as I can see this overhead should be independent of the document counts. &lt;/p&gt;

&lt;p&gt;I have attached all the logs of each experiment ( each collection used different db locations) of the config, mongods and the mongos. The configurations I used is also attached&lt;/p&gt;</description>
                <environment></environment>
        <key id="1062210">SERVER-45265</key>
            <summary>Incosistant sharding overhead</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13204">Community Answered</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="modithadha88@gmail.com">Moditha Hewasinghage</reporter>
                        <labels>
                    </labels>
                <created>Fri, 20 Dec 2019 00:19:28 +0000</created>
                <updated>Fri, 27 Oct 2023 15:56:36 +0000</updated>
                            <resolved>Sun, 22 Dec 2019 12:16:18 +0000</resolved>
                                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2660994" author="modithadha88@gmail.com" created="Mon, 23 Dec 2019 10:57:31 +0000"  >&lt;p&gt;@Dimitry Agranat I think you misunderstood the graph. This bottleneck is the memory when the indexes do not fit in memory. Regardless of the value in 64 million have a look at the graph until 32 million. the gap between sharded and non-sharded at 2 million is way less than when it is 32 million. If the sharding has a constant overhead this should not happen.&lt;/p&gt;</comment>
                            <comment id="2660375" author="dmitry.agranat" created="Sun, 22 Dec 2019 12:15:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=modithadha88%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;modithadha88@gmail.com&quot;&gt;modithadha88@gmail.com&lt;/a&gt;, based on your graph, it looks like you are hitting some bottleneck around 32 million docs, both on sharding and non-sharded instances. So the reported &quot;overhead&quot; looks consistent between both configurations.&lt;/p&gt;

&lt;p&gt;The SERVER project is for bugs and feature suggestions for the MongoDB server. As this ticket does not appear to be a bug, I will now close it. If you need further assistance troubleshooting, I encourage you to ask our community by posting on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or on &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="241041" name="configserver.conf" size="120" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:18:59 +0000"/>
                            <attachment id="241046" name="image-2019-12-20-01-14-53-779.png" size="7336" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:14:54 +0000"/>
                            <attachment id="241040" name="logs.zip" size="656058" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:19:01 +0000"/>
                            <attachment id="241045" name="mongos.conf" size="87" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:18:59 +0000"/>
                            <attachment id="241044" name="shard1.conf" size="379" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:18:59 +0000"/>
                            <attachment id="241043" name="shard2.conf" size="379" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:18:59 +0000"/>
                            <attachment id="241042" name="shard3.conf" size="379" author="modithadha88@gmail.com" created="Fri, 20 Dec 2019 00:18:59 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 22 Dec 2019 12:15:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 7 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 7 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>modithadha88@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwchsf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hw0nfr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwc41r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>