<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:20:02 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9314] cursors that return over 60 million objects are extremely slow</title>
                <link>https://jira.mongodb.org/browse/SERVER-9314</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;It looks like getmores on cursors that return a large number of objects run significantly slower than cursors that return fewer objects.  We noticed this trying to run mongodump on one of our collections which has 84M objects.  collection.stats() returns:&lt;br/&gt;
{&lt;br/&gt;
	&quot;ns&quot; : &quot;data.app_00da3da8-3ec2-490b-ac3a-1ac5d12d0814:SessionEvent&quot;,&lt;br/&gt;
	&quot;count&quot; : 84423082,&lt;br/&gt;
	&quot;size&quot; : 57713849288,&lt;br/&gt;
	&quot;avgObjSize&quot; : 683.6264197035592,&lt;br/&gt;
	&quot;storageSize&quot; : 59682012800,&lt;br/&gt;
	&quot;numExtents&quot; : 49,&lt;br/&gt;
	&quot;nindexes&quot; : 4,&lt;br/&gt;
	&quot;lastExtentSize&quot; : 2146426864,&lt;br/&gt;
	&quot;paddingFactor&quot; : 1,&lt;br/&gt;
	&quot;systemFlags&quot; : 1,&lt;br/&gt;
	&quot;userFlags&quot; : 0,&lt;br/&gt;
	&quot;totalIndexSize&quot; : 8947258432,&lt;br/&gt;
	&quot;indexSizes&quot; : &lt;/p&gt;
{
		&quot;_id_&quot; : 3478397440,
		&quot;_acl_1&quot; : 1572457376,
		&quot;_acl.*.r_1&quot; : 1572457376,
		&quot;_created_at_1&quot; : 2323946240
	}
&lt;p&gt;,&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;


&lt;p&gt;If we try to mongodump this collection it takes about 7 hours.  If we instead dump the collection by parts (i.e. split the _id space into 4 parts) and dump them individually, the total run time is about 1.5 hours.  We have another collection whose on disk size is greater, but with fewer objects which dumps in about 2 hours.  Here is collection.stat() on that collection:&lt;br/&gt;
{&lt;br/&gt;
	&quot;ns&quot; : &quot;data.app_d237a400-f548-42cb-85e3-1643daa0dd4e:SaveGame&quot;,&lt;br/&gt;
	&quot;count&quot; : 1636453,&lt;br/&gt;
	&quot;size&quot; : 114000989904,&lt;br/&gt;
	&quot;avgObjSize&quot; : 69663.46720865188,&lt;br/&gt;
	&quot;storageSize&quot; : 114517589216,&lt;br/&gt;
	&quot;numExtents&quot; : 72,&lt;br/&gt;
	&quot;nindexes&quot; : 7,&lt;br/&gt;
	&quot;lastExtentSize&quot; : 2146426864,&lt;br/&gt;
	&quot;paddingFactor&quot; : 1,&lt;br/&gt;
	&quot;systemFlags&quot; : 1,&lt;br/&gt;
	&quot;userFlags&quot; : 0,&lt;br/&gt;
	&quot;totalIndexSize&quot; : 398171200,&lt;br/&gt;
	&quot;indexSizes&quot; : &lt;/p&gt;
{
		&quot;_id_&quot; : 63372176,
		&quot;UserId_1&quot; : 75538064,
		&quot;_acl_1&quot; : 28305312,
		&quot;_acl.*.r_1&quot; : 28305312,
		&quot;_created_at_1&quot; : 41648544,
		&quot;UDID_1&quot; : 130153744,
		&quot;location_1&quot; : 30848048
	}
&lt;p&gt;,&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;


&lt;p&gt;Experimentally, the point at which performance falls off a cliff is about 60M objects in the result set.&lt;/p&gt;</description>
                <environment>Ubuntu 12.04</environment>
        <key id="71393">SERVER-9314</key>
            <summary>cursors that return over 60 million objects are extremely slow</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="rui.zhang">Rui Zhang</assignee>
                                    <reporter username="charity@parse.com">charity majors</reporter>
                        <labels>
                    </labels>
                <created>Wed, 10 Apr 2013 20:14:52 +0000</created>
                <updated>Wed, 10 Dec 2014 23:19:20 +0000</updated>
                            <resolved>Tue, 10 Jun 2014 21:28:16 +0000</resolved>
                                    <version>2.2.3</version>
                                                    <component>Performance</component>
                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="616159" author="ramon.fernandez" created="Tue, 10 Jun 2014 21:24:32 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charity%40parse.com&quot; class=&quot;user-hover&quot; rel=&quot;charity@parse.com&quot;&gt;charity@parse.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;we haven&apos;t heard back from you for some time, so I&apos;m going to mark this ticket as resolved. If this is still an issue for you, feel free to re-open and provide the additional information that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rui.zhang&quot; class=&quot;user-hover&quot; rel=&quot;rui.zhang&quot;&gt;rui.zhang&lt;/a&gt; requested.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="584127" author="rui.zhang" created="Wed, 14 May 2014 20:44:23 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charity%40parse.com&quot; class=&quot;user-hover&quot; rel=&quot;charity@parse.com&quot;&gt;charity@parse.com&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;Just a friendly reminder. Could you please provide more details on this?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Rui&lt;/p&gt;</comment>
                            <comment id="563063" author="rui.zhang" created="Fri, 25 Apr 2014 16:46:18 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charity%40parse.com&quot; class=&quot;user-hover&quot; rel=&quot;charity@parse.com&quot;&gt;charity@parse.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I tried to reproduce this issue, so far, I haven&apos;t be able to do this with my setup. I have done my test with 80M doc collection, average doc size is ~2k. I did not see significant slowdown by varying dump size between 20M to 80M. &lt;/p&gt;

&lt;p&gt;are you still seeing this issue? if so, could you provide me more details:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;what is the server HW configure for your server, CPU/memory etc&lt;/li&gt;
	&lt;li&gt;are you running from real server or any cloud VM environment&lt;/li&gt;
	&lt;li&gt;if you are running the mongodump, can you attach the log for the mongodump commands for the full and partial dump? If it takes too long, you can just attach the first 20 lines.&lt;/li&gt;
	&lt;li&gt;if you are running the dump command, is there any log in mongod about slow operations, if so, can you attach some samples?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thanks for your help!&lt;br/&gt;
Rui&lt;/p&gt;</comment>
                            <comment id="310708" author="dan@10gen.com" created="Wed, 10 Apr 2013 23:52:09 +0000"  >&lt;p&gt;Thanks for the detailed report; we will attempt to reproduce.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 10 Apr 2013 23:52:09 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 36 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 36 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>charity@parse.com</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>rui.zhang</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmxhb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg2tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7358</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsztx3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>