<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:52:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-340] Getting MongoInternalException &quot;can&apos;t do getmore&quot; while  working on data &gt; 10,000. Dont get the exception when working with lesser amounts of data</title>
                <link>https://jira.mongodb.org/browse/JAVA-340</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt; DBCursor usersiteResult =&lt;br/&gt;
                usersiteCollection.find( userSiteConditionObj );&lt;/p&gt;

&lt;p&gt;            try&lt;br/&gt;
            {&lt;/p&gt;

&lt;p&gt;                while ( usersiteResult.hasNext() )&lt;/p&gt;
{
               //Do operation
            }
&lt;p&gt;catch(MongoInternalException e){&lt;br/&gt;
                } &lt;/p&gt;</description>
                <environment></environment>
        <key id="16505">JAVA-340</key>
            <summary>Getting MongoInternalException &quot;can&apos;t do getmore&quot; while  working on data &gt; 10,000. Dont get the exception when working with lesser amounts of data</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="nivi">Nivi Kumar</reporter>
                        <labels>
                            <label>cursor</label>
                    </labels>
                <created>Fri, 29 Apr 2011 23:08:32 +0000</created>
                <updated>Wed, 11 Sep 2019 19:12:49 +0000</updated>
                            <resolved>Tue, 14 Jun 2011 06:19:02 +0000</resolved>
                                                                    <component>API</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="32572" author="antoine" created="Fri, 13 May 2011 19:23:17 +0000"  >&lt;p&gt;Hi Nivi,&lt;br/&gt;
let me know if this makes sense and I can close ticket.&lt;br/&gt;
thx&lt;/p&gt;</comment>
                            <comment id="31628" author="antoine" created="Mon, 9 May 2011 16:17:39 +0000"  >&lt;p&gt;Nivi,&lt;br/&gt;
the likely issue as Eliot pointed out is that the cursor is timed out on the server.&lt;br/&gt;
This may happen if one of the batches takes more than 10min to process from the client side.&lt;br/&gt;
As such, using a smaller batch size should help, since the time between queries to mongod is smaller.&lt;br/&gt;
To test if that is the issue, you can set Bytes.QUERYOPTION_NOTIMEOUT as a query option and see if you can process all records.&lt;/p&gt;</comment>
                            <comment id="31489" author="eliot" created="Sun, 8 May 2011 02:52:20 +0000"  >&lt;p&gt;The most likely cause is a timed out cursor.&lt;br/&gt;
Are you doing a lot of client side work on these object or just iterating fast?&lt;br/&gt;
If you&apos;re doing client side work, you should make the batch size smaller (100 maybe) and see if that alleviates it.&lt;/p&gt;</comment>
                            <comment id="31412" author="nivi" created="Fri, 6 May 2011 22:15:32 +0000"  >&lt;p&gt;Sorry about responding late. Since we are in run the business mode with this application, it takes a while to debug and respond.&lt;/p&gt;

&lt;p&gt;I am not sure how many records it iterates through before throwing a response. However if I try to get through 10,000 records, it completes around 4000.&lt;br/&gt;
After I used 2.5.3 driver, I did not see the exception any more however it was still not processing all the records.&lt;br/&gt;
I then tried setting the batch size to 500 and then running it. By doing this, it seems like it is processing almost all the records . However it is taking a lot of time to do so. Is this a side effect of using batch size? If it took 100,000 records an hour or two before , now it is taking around 4-5 hours.&lt;/p&gt;</comment>
                            <comment id="30749" author="antoine" created="Tue, 3 May 2011 02:58:33 +0000"  >&lt;p&gt;questions:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;how many docs do you iterate through before you get exception?&lt;/li&gt;
	&lt;li&gt;how long does it take from the time you do find() to the time you get exception (e.g. 2min)?&lt;/li&gt;
	&lt;li&gt;can you run the exact same query in shell and iterate through docs, see if you get error too (do a find(...).forEach(...))&lt;/li&gt;
	&lt;li&gt;in shell, run same command and add .explain() at the end to get information.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;One possible issue is that mongod needs to &quot;scan and order&quot; which has limitations in terms of how many records may be returned.&lt;br/&gt;
thanks&lt;/p&gt;</comment>
                            <comment id="30732" author="nivi" created="Mon, 2 May 2011 23:52:55 +0000"  >&lt;p&gt;I tried with the 2.5.3 driver and I get this exception now&lt;/p&gt;

&lt;p&gt; cursor not found on server&lt;/p&gt;

&lt;p&gt;More info on what the application is trying to do.&lt;br/&gt;
I am trying to iterate through 5000 records. These 5000 records are obtained from another collection. Each record has a network location name. The application then reads the files in that location and stores relevant information in another collection. I used to run the same for over 100000 records and have not faced this issue before.&lt;/p&gt;
</comment>
                            <comment id="30694" author="nivi" created="Mon, 2 May 2011 21:06:56 +0000"  >&lt;p&gt;Thanks Antoine.&lt;/p&gt;

&lt;p&gt;I currently don&apos;t have stack trace enabled in prod. This is the message in&lt;br/&gt;
log. I will try and run in test to get the full stack trace .&lt;/p&gt;

&lt;p&gt;As for the driver , we are using mongo-2.0.jar. I will test with the latest&lt;br/&gt;
driver and get back to you.&lt;/p&gt;

&lt;p&gt;Cheers&lt;br/&gt;
nivi&lt;/p&gt;




</comment>
                            <comment id="30669" author="antoine" created="Mon, 2 May 2011 18:49:32 +0000"  >&lt;p&gt;Hi Nivi,&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;could you give the exact exception you are getting, including possible stack trace.&lt;/li&gt;
	&lt;li&gt;what driver version are you using? If older than 2.5.3, is it possible for you to test with latest driver in testing environment?&lt;br/&gt;
thanks&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="30645" author="nivi" created="Mon, 2 May 2011 17:37:55 +0000"  >&lt;p&gt;Any suggestions to overcome this issue?&lt;/p&gt;</comment>
                            <comment id="30508" author="nivi" created="Sat, 30 Apr 2011 17:35:02 +0000"  >&lt;p&gt;usersitecondition obj is the query object user to query the user site collection.&lt;/p&gt;

&lt;p&gt;   DBObject userSiteRecordsCondition =&lt;br/&gt;
            new BasicDBObject( MONGO_OPERATION_GTE, minUsrRecords );&lt;br/&gt;
        userSiteRecordsCondition.put( MONGO_OPERATION_LTE, maxUsrRecords );&lt;/p&gt;

&lt;p&gt;        userSiteConditionObj.put( MongoDBCollectionFieldKeys.KEY_COUNTER_ID,&lt;br/&gt;
            userSiteRecordsCondition );&lt;/p&gt;

&lt;p&gt;The query is to get all records in the user site collection based on the minimum and maximum counter id that the user specifies. If min is 10 and max is 100, the query would get all the 90 records and iterate over it. For 10000 or less records it seems to be fine . However when it is more than that it throws the &quot;cant get more exception&quot; &lt;/p&gt;</comment>
                            <comment id="30455" author="eliot" created="Sat, 30 Apr 2011 04:39:08 +0000"  >&lt;p&gt;What are the contents of userSiteConditionObj&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrhc67:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14772</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>