<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:29:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32200] ISSUE: Wrong Document Count returning from MONGODBServer</title>
                <link>https://jira.mongodb.org/browse/SERVER-32200</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;&lt;ins&gt;Description:&lt;/ins&gt;&lt;/b&gt;&lt;br/&gt;
We are using MongoDB Server version 3.4 &lt;br/&gt;
We have a collection with 5.5 Million=(5482968) documents and indexSize=283.6MB . we have set the RAM Size as 1.1GB&lt;br/&gt;
When we try to retrieve the document count from MongoDBServer taking long time to respond and finally returning wrong count &lt;br/&gt;
we have created indexes on 8 fields on this collection .Our Collection Size =7.8GB&lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;ins&gt;Issue we are facing:&lt;/ins&gt;&lt;/b&gt;&lt;br/&gt;
1. Some times Wrong document count is returned from MongoDBServer&lt;br/&gt;
	Actual Count Returned from MonogoDBServer=5080966 &lt;br/&gt;
	Expected Count Returned from MonogoDBServer=5482968 &lt;br/&gt;
2. Some times MongoDBServer taking too long time to respond &lt;span class=&quot;error&quot;&gt;&amp;#91;in minutes&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;ins&gt;Note:&lt;/ins&gt;&lt;/b&gt; In production, this collection can grow to have more documents (Billions of documents)&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;Requesting you to Kindly address our below queries  ,&lt;/ins&gt;&lt;br/&gt;
a)Whether increasing the RAM Size will address our above Mentioned issues&lt;span class=&quot;error&quot;&gt;&amp;#91;count &amp;#93;&lt;/span&gt;&lt;br/&gt;
b)Will reducing number of indexed columns/fields help in addressing these issues?&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;How to Reproduce:&lt;/ins&gt;&lt;/p&gt;

&lt;p&gt;1.Create a collection say &apos;testCollection&apos; with 5.5Million or more document with average document size of 1.5KiB&lt;br/&gt;
2.Create index for 8 fields ensure that indexSize grows to &amp;gt;=280MB&lt;br/&gt;
3.Set the mongoDB RAMSize to 1.1GB&lt;br/&gt;
*db.adminCommand(&lt;/p&gt;
{setParameter: 1, internalQueryExecMaxBlockingSortBytes: 1073741824}
&lt;p&gt;)*&lt;br/&gt;
4.Perform db.testCollection.count(with some condition on a field)&lt;/p&gt;
</description>
                <environment></environment>
        <key id="469204">SERVER-32200</key>
            <summary>ISSUE: Wrong Document Count returning from MONGODBServer</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.agarunov">Mark Agarunov</assignee>
                                    <reporter username="Urs">Sandeep [X]</reporter>
                        <labels>
                    </labels>
                <created>Thu, 7 Dec 2017 13:02:37 +0000</created>
                <updated>Sun, 7 Jan 2018 04:36:18 +0000</updated>
                            <resolved>Fri, 8 Dec 2017 20:07:54 +0000</resolved>
                                    <version>3.4.0</version>
                                                    <component>Index Maintenance</component>
                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1751629" author="urs" created="Thu, 14 Dec 2017 04:54:55 +0000"  >&lt;p&gt;Dear Mark,&lt;/p&gt;

&lt;p&gt;Please let us know in case new ticket needs to be opened here ...&lt;/p&gt;

&lt;p&gt;Thanks &amp;amp; Regards&lt;br/&gt;
Sandeep KS&lt;/p&gt;</comment>
                            <comment id="1750493" author="urs" created="Wed, 13 Dec 2017 05:29:01 +0000"  >&lt;p&gt;Dear Mark,&lt;br/&gt;
I am also attaching the system.profile output for your analysis &lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;ins&gt;Note:&lt;/ins&gt;&lt;/b&gt;&lt;br/&gt;
We recorded the query by setting time as 4 minutes&lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;#39;240000&amp;#39;&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/173354/173354_systemprofile_gt-4minutes.zip&quot; title=&quot;systemprofile_gt-4minutes.zip attached to SERVER-32200&quot;&gt;systemprofile_gt-4minutes.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;Thanks &amp;amp; Regards&lt;br/&gt;
Sandeep KS &lt;/p&gt;</comment>
                            <comment id="1749662" author="urs" created="Tue, 12 Dec 2017 15:05:51 +0000"  >&lt;p&gt;Dear Mark,&lt;/p&gt;

&lt;p&gt;Thank you for your inputs!!&lt;br/&gt;
 Based on your inputs,&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;We Executed validate command on the collection, later we performed explain() on the DB Query and observed that the SORT stage with &apos;timestamp&apos; field as index was inside the &apos;RejectionPlan&apos;.&lt;/li&gt;
	&lt;li&gt;Now we modified index creation so that the SORT stage with &apos;timestamp&apos; field as index came inside the &apos;WinningPlan&apos;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;ins&gt;&lt;b&gt;Queries:&lt;/b&gt;&lt;/ins&gt;&lt;br/&gt;
&lt;ins&gt;&lt;b&gt;Query1:&lt;/b&gt;&lt;/ins&gt; &lt;font color=&quot;red&quot;&gt;How ever , Still we observe that execution of queries from mongoserver is &lt;ins&gt;taking time in minutes [ &amp;gt;15 minutes!!!]&lt;/ins&gt;&lt;/font&gt;&lt;br/&gt;
We have attached below logs for your analysis &lt;br/&gt;
&lt;ins&gt;&lt;b&gt;a)Query plan Output[]&lt;/b&gt;&lt;/ins&gt;&lt;br/&gt;
	FileName:QueryPlan-Explain-output-12122017&lt;br/&gt;
*&lt;ins&gt;&lt;b&gt;b)db.currentOp Operation output *&lt;/ins&gt;&lt;/b&gt;&lt;br/&gt;
	FileName:currentOp-output-12122017&lt;br/&gt;
	Note: we waited for 5 minutes and we took the &apos;currentOp&apos; output&lt;br/&gt;
*&lt;ins&gt;&lt;b&gt;c) log and diagnostic.data log files&lt;/ins&gt;&lt;/b&gt;&lt;br/&gt;
Also attached fresh db-&amp;gt;log and db-&amp;gt;diagnostic.data  * for your analysis &lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;ins&gt;Query2:&lt;/ins&gt;&lt;/b&gt;&lt;font color=&quot;red&quot;&gt; We are not sure why other index which we created went to rejection Plan?&lt;/font&gt;&lt;br/&gt;
 Example : We created &apos;deviceId&apos; field as one of the index, only after removal of this index &lt;br/&gt;
  then only the &apos;timestampIdx&apos; ,went to winningPlan&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/173279/173279_SERVER-32200.zip&quot; title=&quot;SERVER-32200.zip attached to SERVER-32200&quot;&gt;SERVER-32200.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;Thanks &amp;amp; Regards&lt;br/&gt;
Sandeep KS&lt;/p&gt;
</comment>
                            <comment id="1747303" author="mark.agarunov" created="Fri, 8 Dec 2017 20:07:35 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Urs&quot; class=&quot;user-hover&quot; rel=&quot;Urs&quot;&gt;Urs&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report. After looking over this, the incorrect counts may be due to previous unclean shutdowns. This can be fixed by running the &lt;a href=&quot;https://docs.mongodb.com/v3.4/reference/method/db.collection.validate/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validate command&lt;/a&gt; on the collection. &lt;/p&gt;

&lt;p&gt;Additional memory will likely improve performance. In addition to increasing the cache size, more memory would allow sorts to fit in memory and  not have to be written to disk, causing a significant performance hit. Looking over the logs, it appears that the queries are not effectively using indexes, which is likely the cause of the long run times for these queries. Additionally, it appears that the Indexes that did exist for the table were dropped just before the queries started to get slower. In addition to using indexes for these queries, I would recommend &lt;a href=&quot;https://docs.mongodb.com/v3.4/release-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upgrading MongoDB to at least the latest 3.4 release&lt;/a&gt;, as there have been many performance improvements in more recent versions, especially relating to the aggregation framework. &lt;/p&gt;

&lt;p&gt;As I do not see anything in the logs or diagnostic data to indicate a bug in MongoDB, I&apos;ve closed this ticket. If additional information comes to light, or you are still seeing slow performance after upgrading and making sure the queries effectively use the index, please let us know and we can reopen the ticket.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1746558" author="urs" created="Fri, 8 Dec 2017 07:07:24 +0000"  >&lt;p&gt;Hello Mark,&lt;/p&gt;

&lt;p&gt;Thanks for your support, kindly find archived $dbpath/diagnostic.data directory &amp;amp; log.zip attached to this ticket.&lt;br/&gt;
Let us know if anything else is required.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Sandeep&lt;/p&gt;</comment>
                            <comment id="1745626" author="mark.agarunov" created="Thu, 7 Dec 2017 17:28:10 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Urs&quot; class=&quot;user-hover&quot; rel=&quot;Urs&quot;&gt;Urs&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report. To get a better idea of why this may be happening, could you please provide the following:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The complete logs from all affected &lt;tt&gt;mongod&lt;/tt&gt; nodes&lt;/li&gt;
	&lt;li&gt;Please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory and attach it to this ticket.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This should give some insight into why the count is inaccurate and taking a long time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="173279" name="SERVER-32200.zip" size="511519" author="Urs" created="Tue, 12 Dec 2017 14:51:40 +0000"/>
                            <attachment id="172780" name="diagnostic.data.zip" size="56687835" author="Urs" created="Fri, 8 Dec 2017 06:57:32 +0000"/>
                            <attachment id="172779" name="log.zip" size="12280357" author="Urs" created="Fri, 8 Dec 2017 06:54:53 +0000"/>
                            <attachment id="173354" name="systemprofile_gt-4minutes.zip" size="18695" author="Urs" created="Wed, 13 Dec 2017 05:27:45 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 7 Dec 2017 17:28:10 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 8 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 8 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>mark.agarunov</customfieldvalue>
            <customfieldvalue>Urs</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htlrnr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htdcl3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htldrr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>