[SERVER-4111] seems like explain() loads documents, prevents doing just index preheat Created: 20/Oct/11  Updated: 30/Mar/12  Resolved: 21/Oct/11

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Antoine Girbal Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

the line we usually use for index preheat seems to load up all docs.
In this example docs are created like:

> var inc = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
> var str = inc
> while (str.length < 2000) str += inc;
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> for (var i = 0; i < 200000; ++i) db.large.save(

{i: i, s: str}

);
> db.large.ensureIndex(

{i: 1}

);
> db.large.find(

{i:1}

).explain()
{
"cursor" : "BtreeCursor i_1",
"nscanned" : 1,
"nscannedObjects" : 1,
"n" : 1,
"millis" : 0,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" :

{ "i" : [ [ 1, 1 ] ] }

}

// create index so that we can find docs without table scan
> db.large.find({s: {$ne: "a"} }).explain()
{
"cursor" : "BasicCursor",
"nscanned" : 200000,
"nscannedObjects" : 200000,
"n" : 200000,
"millis" : 106,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {

}
}

Then if you load up all docs, resident is 445m
1054 antoine 20 0 1215m 445m 428m S 0 2.8 0:05.23 mongod

Now restart mongod, and query docs 1 by 1 with explain:
> for (var i = 0; i < 200000; i += 1) db.large.find(

{i: i}

).explain()
{
"cursor" : "BtreeCursor i_1",
"nscanned" : 1,
"nscannedObjects" : 1,
"n" : 1,
"millis" : 0,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" :

{ "i" : [ [ 199999, 199999 ] ] }

}

Resident is low, only index was touched:
2146 antoine 20 0 1145m 36m 25m S 0 0.2 0:44.24 mongod

Now use the trick we usually do for index preheat:
> db.large.find({i: {$ne: -1} }).explain()
{
"cursor" : "BtreeCursor i_1 multi",
"nscanned" : 200000,
"nscannedObjects" : 200000,
"n" : 200000,
"millis" : 2980,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {
"i" : [
[

{ "$minElement" : 1 }

,
-1
],
[
-1,

{ "$maxElement" : 1 }

]
]
}
}

Result is that all docs / pages are loaded in resident:
2146 antoine 20 0 1145m 432m 421m S 0 2.7 0:45.05 mongod

It seems as thought the "BTree Multi" touches actual docs, at least in the explain.



 Comments   
Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

Already is pre-heat case SERVER-2023

Comment by Antoine Girbal [ 21/Oct/11 ]

Ok this works well:
> db.large.find({},

{i: 1, _id: 0}

).hint(

{i: 1}

).count()
200000

changing this ticket to add a preheat method, seems to come up pretty often.
will create ticket for $ne case.

Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

$ne could be special cased for multi-key, but i would make a clean case for that.

Easiest way to preheat an index is

db.foo.find( {} ,

{ <index keys> , _id : 0 }

).hint( <index keys> ).count()

Comment by Antoine Girbal [ 21/Oct/11 ]
  • in this simple case it's all integer values, why would it need to access docs?
  • what is an example of case when it needs to access document? Here multikey is false.
  • is there another way to warm up just index?
    thanks
Comment by Eliot Horowitz (Inactive) [ 21/Oct/11 ]

This is because we can't check $ne just using index in many cases.

Generated at Thu Feb 08 03:04:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.