[SERVER-9314] cursors that return over 60 million objects are extremely slow Created: 10/Apr/13  Updated: 10/Dec/14  Resolved: 10/Jun/14

Status: Closed
Project: Core Server
Component/s: Performance, Querying
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: charity majors Assignee: Rui Zhang (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04


Operating System: ALL
Participants:

 Description   

It looks like getmores on cursors that return a large number of objects run significantly slower than cursors that return fewer objects. We noticed this trying to run mongodump on one of our collections which has 84M objects. collection.stats() returns:
{
"ns" : "data.app_00da3da8-3ec2-490b-ac3a-1ac5d12d0814:SessionEvent",
"count" : 84423082,
"size" : 57713849288,
"avgObjSize" : 683.6264197035592,
"storageSize" : 59682012800,
"numExtents" : 49,
"nindexes" : 4,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 8947258432,
"indexSizes" :

{ "_id_" : 3478397440, "_acl_1" : 1572457376, "_acl.*.r_1" : 1572457376, "_created_at_1" : 2323946240 }

,
"ok" : 1
}

If we try to mongodump this collection it takes about 7 hours. If we instead dump the collection by parts (i.e. split the _id space into 4 parts) and dump them individually, the total run time is about 1.5 hours. We have another collection whose on disk size is greater, but with fewer objects which dumps in about 2 hours. Here is collection.stat() on that collection:
{
"ns" : "data.app_d237a400-f548-42cb-85e3-1643daa0dd4e:SaveGame",
"count" : 1636453,
"size" : 114000989904,
"avgObjSize" : 69663.46720865188,
"storageSize" : 114517589216,
"numExtents" : 72,
"nindexes" : 7,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 398171200,
"indexSizes" :

{ "_id_" : 63372176, "UserId_1" : 75538064, "_acl_1" : 28305312, "_acl.*.r_1" : 28305312, "_created_at_1" : 41648544, "UDID_1" : 130153744, "location_1" : 30848048 }

,
"ok" : 1
}

Experimentally, the point at which performance falls off a cliff is about 60M objects in the result set.



 Comments   
Comment by Ramon Fernandez Marina [ 10/Jun/14 ]

Hi charity@parse.com,

we haven't heard back from you for some time, so I'm going to mark this ticket as resolved. If this is still an issue for you, feel free to re-open and provide the additional information that rui.zhang requested.

Regards,
Ramón.

Comment by Rui Zhang (Inactive) [ 14/May/14 ]

Hi charity@parse.com,

Just a friendly reminder. Could you please provide more details on this?

Thanks,
Rui

Comment by Rui Zhang (Inactive) [ 25/Apr/14 ]

Hi charity@parse.com

I tried to reproduce this issue, so far, I haven't be able to do this with my setup. I have done my test with 80M doc collection, average doc size is ~2k. I did not see significant slowdown by varying dump size between 20M to 80M.

are you still seeing this issue? if so, could you provide me more details:

  1. what is the server HW configure for your server, CPU/memory etc
  2. are you running from real server or any cloud VM environment
  3. if you are running the mongodump, can you attach the log for the mongodump commands for the full and partial dump? If it takes too long, you can just attach the first 20 lines.
  4. if you are running the dump command, is there any log in mongod about slow operations, if so, can you attach some samples?

Thanks for your help!
Rui

Comment by Daniel Pasette (Inactive) [ 10/Apr/13 ]

Thanks for the detailed report; we will attempt to reproduce.

Generated at Thu Feb 08 03:20:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.