[SERVER-14142] Runner error: Overflow Created: 03/Jun/14  Updated: 10/Dec/14  Resolved: 08/Jul/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: flyinflash Assignee: Thomas Rueckstiess
Resolution: Done Votes: 1
Labels: bug, driver
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04 64bit
mongoDB server 2.6.1


Operating System: Linux
Participants:

 Description   

When I do query

qs = MyModel.objects.all()

It always raises exception

pymongo.errors.OperationFailure: database error: Runner error: Overflow sort stage buffered data usage of 33555614 bytes exceeds internal limit of 33554432 bytes



 Comments   
Comment by Thomas Rueckstiess [ 08/Jul/14 ]

Hi, from the information you provided, we can now see that the query was on the movie.movie collection of the form:

{ 
    $query: { 
        $or: [ 
            { artist_list: [ "John" ] }, 
            { genre_list: [] }, 
            { manufacturer_list: [] } 
        ] 
    }, 
    $orderby: { 
        created_ts: -1 
    } 
}

There is an index present on created_ts, but it is a sparse index.

{
    "key": {
        "created_ts": -1
    },
    "name": "created_ts_-1",
    "ns": "movie.movie",
    "sparse": true,
    "v": 1
},

The planner did not use this index, probably because the result set was not entirely present in the index:

planSummary: COLLSCAN ntoreturn:0 keyUpdates:0 exception: Runner error: Overflow sort stage buffered data usage of 33554724 bytes exceeds internal limit of 33554432 bytes code:17144 numYields:1 locks(micros) r:627300 reslen:158 519ms

This is a new behavior in 2.6, documented under Sparse Indexes.

If a sparse index would result in an incomplete result set for queries and sort operations, MongoDB will not use that index unless a hint() explicitly specifies the index.

You can either re-create the index to be non-sparse (the default), or use an explicit hint to force the use of that index. However, you may get incomplete results in the latter case.

I hope this answers your question. I'm closing this ticket as "works as designed".

Regards,
Thomas

Comment by flyinflash [ 29/Jun/14 ]

debian(mongod-2.6.3) movie> db.movie.getIndexes()
[
  {
    "key": {
      "_id": 1
    },
    "name": "_id_",
    "ns": "movie.movie",
    "v": 1
  },
  {
    "key": {
      "created_ts": -1
    },
    "name": "created_ts_-1",
    "ns": "movie.movie",
    "sparse": true,
    "v": 1
  },
  {
    "key": {
      "pub_date": -1
    },
    "name": "pub_date_-1",
    "ns": "movie.movie",
    "sparse": true,
    "v": 1
  },
  {
    "key": {
      "link_list_length": -1
    },
    "name": "link_list_length_-1",
    "ns": "movie.movie",
    "sparse": true,
    "v": 1
  },
  {
    "key": {
      "pic_list_length": -1
    },
    "name": "pic_list_length_-1",
    "ns": "movie.movie",
    "sparse": true,
    "v": 1
  },
  {
    "key": {
      "product_id": -1
    },
    "name": "product_id_-1",
    "ns": "movie.movie",
    "sparse": true,
    "unique": true,
    "v": 1
  }
]

Comment by Thomas Rueckstiess [ 26/Jun/14 ]

Hi flyinflash, could you please provide the list of indexes on the movie.movie collection? I'd like to see if there is an appropriate index that could support the sort.

You can do that with:

use movie
db.movie.getIndexes()

Please attach the output here as a comment.

Thanks,
Thomas

Comment by J Rassi [ 26/Jun/14 ]

I've re-opened the issue.

Could you also please provide the output of running db.movie.getIndexes()?

Comment by flyinflash [ 26/Jun/14 ]

It seems I can't reopen this issue.

Comment by flyinflash [ 26/Jun/14 ]

MongoDB shell version: 2.6.3

ubuntu 12.04 LTS

2014-06-26T23:32:50.393+0800 [conn1411861] ERROR: Runner error, stats:
{ "type" : "SORT",
  "works" : 82092,
  "yields" : 641,
  "unyields" : 641,
  "invalidates" : 0,
  "advanced" : 0,
  "needTime" : 82090,
  "needFetch" : 0,
  "isEOF" : 0,
  "forcedFetches" : 0,
  "memUsage" : 33554724,
  "memLimit" : 33554432,
  "children" : [ 
    { "type" : "COLLSCAN",
      "works" : 82090,
      "yields" : 641,
      "unyields" : 641,
      "invalidates" : 0,                                                                                                                                                         
      "advanced" : 11582,                                                                                                                                                        
      "needTime" : 70508,                                                                                                                                                        
      "needFetch" : 0,                                                                                                                                                           
      "isEOF" : 0,
      "docsTested" : 82089,
      "children" : [] } ] }
2014-06-26T23:32:50.395+0800 [conn1411861] assertion 17144 Runner error: Overflow sort stage buffered data usage of 33554724 bytes exceeds internal limit of 33554432 bytes ns:movie.movie query:{ $query: { $or: [ { artist_list: [ "John" ] }, { genre_list: [] }, { manufacturer_list: [] } ] }, $orderby: { created_ts: -1 } }
2014-06-26T23:32:50.395+0800 [conn1411861] query movie.movie query: { $query: { $or: [ { artist_list: [ "John" ] }, { genre_list: [] }, { manufacturer_list: [] } ] }, $orderby: { created_ts: -1 } } planSummary: COLLSCAN ntoreturn:0 keyUpdates:0 exception: Runner error: Overflow sort stage buffered data usage of 33554724 bytes exceeds internal limit of 33554432 bytes code:17144 numYields:1 locks(micros) r:627300 reslen:158 519ms

Comment by J Rassi [ 06/Jun/14 ]

I'm resolving this ticket as "Cannot Reproduce". Please re-open it with the requested information if you continue to encounter this issue.

Comment by J Rassi [ 03/Jun/14 ]

In addition to the server log output, could you also provide the output of running db.<collection>.getIndexes() at the shell (replacing <collection> with the name of the collection used here in your model)?

Comment by Bernie Hackett [ 03/Jun/14 ]

Hi, this error is from the server. Can you provide the server logs for the time period this error occurred?

Generated at Thu Feb 08 03:33:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.