[SERVER-7638] explain() returns different results than count() when using hint() Created: 13/Nov/12  Updated: 15/Feb/13  Resolved: 13/Nov/12

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Querying
Affects Version/s: 2.2.1
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: William Miskovetz Assignee: Aaron Staple
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Amazon EC2 running Amazon Linux.


Issue Links:
Duplicate
duplicates SERVER-3372 Allow indexing fields of arbitrary le... Closed
Participants:

 Description   

Posted to mongo users group also. Looking through the existing bugs I found
a bug which sounded similar which was that hint() didn't work with count(). There was a vague comment that the results returned when using hint() would not be valid, but nothing more. I'll look up the bug number after this.

I'm attempting to isolate a performance problem with some PHP code and MongoDB.
I'm beginning to think that either hint() or explain() doesn't do what I think it should do.
In the PHP below, I use to not have the hint. It was painfully slow. Putting the
hint in made things quicker, but the results don't seem correct. I'm sure I'm missing
something, but have a read. Thanks. MongoDB version 2.2.1.

I'm doing the following in PHP:
$results = $db->Results->find(array('Test Results' => array('$exists' => true), 'atSchool' => 'yes', 'onNetwork' => 'yes', 'schoolID' => (string)$s['_id']), array('_id' => 1));
$results = $results->hint(array("Test Results" => 1, "atSchool" => 1,
"onNetwork" => 1, "schoolID" => 1));
$numR = 0;
foreach ($results as $r)

{ $numR = $numR + 1; }

or in mongo shell:
db.Results.find( { "Test Results" :

{ "$exists" : true }

, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint(

{"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}

).explain()

Now here's the issue. explain() returns n=113, which if I read the document correctly should be the number of results returned.
Yet if I run the above mongo shell command with count() instead of explain it returns 5641, which I believe is the correct number.
The PHP code seems to return the same as explain().

explain() output:
{
"cursor" : "BtreeCursor Test Results_1_atSchool_1_onNetwork_1_schoolID_1",
"isMultiKey" : false,
"n" : 113,
"nscannedObjects" : 177,
"nscanned" : 6531,
"nscannedObjectsAllPlans" : 177,
"nscannedAllPlans" : 6531,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 50,
"indexBounds" : {
"Test Results" : [
[

{ "$minElement" : 1 }

,

{ "$maxElement" : 1 }

]
],
"atSchool" : [
[
"yes",
"yes"
]
],
"onNetwork" : [
[
"yes",
"yes"
]
],
"schoolID" : [
[
"550627000691",
"550627000691"
]
]
},
"server" : "ip-1-1-1-1:27017"
}

count() output:
> db.Results.find( { "Test Results" :

{ "$exists" : true }

, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint(

{"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}

).count()
5641

If I check the index I have:
[
{
"v" : 1,
"key" :

{ "Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1 }

,
"ns" : "Edu.Results",
"name" : "Test Results_1_atSchool_1_onNetwork_1_schoolID_1"
}
]

I've rebuilt the indexes, run ensureIndex, etc. No changes. Anyone have ideas?



 Comments   
Comment by William Miskovetz [ 13/Nov/12 ]

Oops, I lied. Forgot /var/log/mongo/mongod.log doesn't have my logs, they are in /data/db/mongod.log.
Sure enough:

Tue Oct 30 05:54:15 [conn4579] warning: not all entries were added to the index, probably some keys were too large
Tue Nov 13 00:49:29 [conn152] warning: not all entries were added to the index, probably some keys were too large

Comment by William Miskovetz [ 13/Nov/12 ]

Thanks. No log messages since I rebuilt the indexes today from the mongo shell and both the log file is empty and no errors reported in the shell. Just FYI.

I'll change my index to not include Test Results. But I'm a bit concerned that I could do a query and get results instead of any type of error and have those results be incorrect. An error is far preferable to bad/wrong results.

Thanks for the debug, fun problem

Comment by Aaron Staple [ 13/Nov/12 ]

Hi William - There's a combination of issues going on here.

The first is that mongo has a limit on the size of a key that can be stored in a btree. Many of your documents have very large "Test Results" objects that cannot be stored in a btree. As a result, these documents will not be returned when a btree index is used. You may see log messages warning you of this fact, eg

Mon Nov 12 22:36:22.666 [conn7] not all entries were added to the index, probably some keys were too large

SERVER-3372 is the ticket for addressing this.

The other relevant ticket relates to index choice. Currently if there are no index bounds on the first field of an index, that index will not be used automatically and must be hinted explicitly. In this case $exists:true is equivalent to no index bounds. This is why you see different results when hinting your index vs not using a hint. Without a hint no index is used, equivalent to hinting $natural:1. The ticket for revisiting this policy is SERVER-2140.

Comment by William Miskovetz [ 13/Nov/12 ]

 
> ids = db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }, { _id:1 } ).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).toArray();
[
        {
                "_id" : ObjectId("5086ccab27c133c32b000018")
        },
        {
                "_id" : ObjectId("5089a49d27c133915300002f")
        },
        {
                "_id" : ObjectId("5089484527c133d451000022")
        },
        {
                "_id" : ObjectId("507d716327c1337c44000000")
        },
        {
                "_id" : ObjectId("507ef61827c133e44e000000")
        },
        {
                "_id" : ObjectId("5080405b27c133cc69000010")
        },
        {
                "_id" : ObjectId("507d84c127c133eb44000014")
        },
        {
                "_id" : ObjectId("509173975ce2643ad27000d7")
        },
        {
                "_id" : ObjectId("507d71b527c133344400002d")
        },
        {
                "_id" : ObjectId("508e8ae827c133410300000c")
        },
        {
                "_id" : ObjectId("50917cc05ce2643ad270033e")
        },
        {
                "_id" : ObjectId("508fe64627c133bb0e000002")
        },
        {
                "_id" : ObjectId("5080101527c133d665000000")
        },
        {
                "_id" : ObjectId("5087f5d027c1336f35000011")
        },
        {
                "_id" : ObjectId("5087f5b627c1339e35000006")
        },
        {
                "_id" : ObjectId("508047e127c133766a000012")
        },
        {
                "_id" : ObjectId("5089448f27c133aa52000010")
        },
        {
                "_id" : ObjectId("5087f65027c133673500000e")
        },
        {
                "_id" : ObjectId("507f03e327c133dd4f000008")
        },
        {
                "_id" : ObjectId("508962ce27c1336f51000057")
        },
        {
                "_id" : ObjectId("508552e527c133952000000b")
        },
        {
                "_id" : ObjectId("507f003a27c133654f00000b")
        },
        {
                "_id" : ObjectId("508adeba27c133b461000030")
        },
        {
                "_id" : ObjectId("508152a027c133a17000000c")
        },
        {
                "_id" : ObjectId("508adf7c27c1339a61000028")
        },
        {
                "_id" : ObjectId("508acf2b27c1338d6100001d")
        },
        {
                "_id" : ObjectId("5080101627c1330365000015")
        },
        {
                "_id" : ObjectId("509128ac27c1338516000000")
        },
        {
                "_id" : ObjectId("50883dab27c1331a4d00000b")
        },
        {
                "_id" : ObjectId("507ef5f027c133664e000020")
        },
        {
                "_id" : ObjectId("5085647627c133b720000022")
        },
        {
                "_id" : ObjectId("50883dba27c133964200001c")
        },
        {
                "_id" : ObjectId("50917d465ce2643ad2700355")
        },
        {
                "_id" : ObjectId("5091289727c133791400000a")
        },
        {
                "_id" : ObjectId("508560c527c1336321000000")
        },
        {
                "_id" : ObjectId("508ace2427c1338961000018")
        },
        {
                "_id" : ObjectId("50855e3827c133f320000005")
        },
        {
                "_id" : ObjectId("508ace1227c133a861000027")
        },
        {
                "_id" : ObjectId("5089491427c1338e5200001f")
        },
        {
                "_id" : ObjectId("50899c4027c1337f53000017")
        },
        {
                "_id" : ObjectId("507dabf327c1337d46000009")
        },
        {
                "_id" : ObjectId("50899bc227c1338a53000008")
        },
        {
                "_id" : ObjectId("5089482627c1338e5200001e")
        },
        {
                "_id" : ObjectId("5089956027c1338853000004")
        },
        {
                "_id" : ObjectId("5089484827c133aa52000018")
        },
        {
                "_id" : ObjectId("507dabf427c133c646000001")
        },
        {
                "_id" : ObjectId("50883dcd27c133ce4c000015")
        },
        {
                "_id" : ObjectId("5087f64027c1339735000014")
        },
        {
                "_id" : ObjectId("508564cf27c1335e2100000a")
        },
        {
                "_id" : ObjectId("50883dbe27c1331e4d00000f")
        },
        {
                "_id" : ObjectId("507dac2a27c1336346000004")
        },
        {
                "_id" : ObjectId("508948db27c1338f5200001c")
        },
        {
                "_id" : ObjectId("5086cb4527c133a12b000022")
        },
        {
                "_id" : ObjectId("507ef5d227c133b24e000011")
        },
        {
                "_id" : ObjectId("5087ffcd27c1338530000027")
        },
        {
                "_id" : ObjectId("5091288827c1339213000004")
        },
        {
                "_id" : ObjectId("507dabfc27c1332d46000028")
        },
        {
                "_id" : ObjectId("508e8ac827c133450300000c")
        },
        {
                "_id" : ObjectId("50901e655ce2643ad26fec3d")
        },
        {
                "_id" : ObjectId("50898b5827c1338d52000078")
        },
        {
                "_id" : ObjectId("50803ead27c1332f6a000005")
        },
        {
                "_id" : ObjectId("507d79db27c1333444000037")
        },
        {
                "_id" : ObjectId("507dabed27c1336446000004")
        },
        {
                "_id" : ObjectId("5089825a27c133a652000056")
        },
        {
                "_id" : ObjectId("507ef5f227c133cc4e000003")
        },
        {
                "_id" : ObjectId("508046f827c133776a00000c")
        },
        {
                "_id" : ObjectId("5080105127c133d365000001")
        },
        {
                "_id" : ObjectId("508565c227c133522100000d")
        },
        {
                "_id" : ObjectId("507dabf227c1339b4600000a")
        },
        {
                "_id" : ObjectId("5091288627c1338f1300000c")
        },
        {
                "_id" : ObjectId("507ef5f127c1339d4e000026")
        },
        {
                "_id" : ObjectId("509024ab5ce2643ad26fec97")
        },
        {
                "_id" : ObjectId("50883dd427c133dd3600005b")
        },
        {
                "_id" : ObjectId("507dabf127c133c646000000")
        },
        {
                "_id" : ObjectId("5080100e27c133d864000021")
        },
        {
                "_id" : ObjectId("508046e427c133cc6900001d")
        },
        {
                "_id" : ObjectId("5080104627c133c865000001")
        },
        {
                "_id" : ObjectId("50917e285ce2643ad2700376")
        },
        {
                "_id" : ObjectId("508046e127c133186a00001f")
        },
        {
                "_id" : ObjectId("509024265ce2643ad26fec89")
        },
        {
                "_id" : ObjectId("50901ddd27c133e70f000008")
        },
        {
                "_id" : ObjectId("508046c327c133d769000018")
        },
        {
                "_id" : ObjectId("5080473627c133f269000028")
        },
        {
                "_id" : ObjectId("507ef6ba27c133ce4e000004")
        },
        {
                "_id" : ObjectId("507ef5ca27c133dd4e000000")
        },
        {
                "_id" : ObjectId("50901de85ce2643ad26fec35")
        },
        {
                "_id" : ObjectId("507ef5e827c133d24e000005")
        },
        {
                "_id" : ObjectId("50901e4a5ce2643ad26fec3b")
        },
        {
                "_id" : ObjectId("509128ce27c1337314000009")
        },
        {
                "_id" : ObjectId("507d6b4b27c1335044000008")
        },
        {
                "_id" : ObjectId("507d714e27c1338444000002")
        },
        {
                "_id" : ObjectId("507ef6a327c133d24e000008")
        },
        {
                "_id" : ObjectId("5080102c27c133dc65000000")
        },
        {
                "_id" : ObjectId("507d739627c1337d44000004")
        },
        {
                "_id" : ObjectId("507d79ff27c1339144000006")
        },
        {
                "_id" : ObjectId("507d6cac27c133344400001c")
        },
        {
                "_id" : ObjectId("507d7a2427c1338444000013")
        },
        {
                "_id" : ObjectId("507d7a8727c1338444000014")
        },
        {
                "_id" : ObjectId("507d7a1a27c133bb44000007")
        },
        {
                "_id" : ObjectId("508046cb27c133186a00001e")
        },
        {
                "_id" : ObjectId("507d6b5327c1333444000019")
        },
        {
                "_id" : ObjectId("507d7a7d27c133904400000f")
        },
        {
                "_id" : ObjectId("507d79e327c1337c44000015")
        },
        {
                "_id" : ObjectId("507d6b8427c133c43600002c")
        },
        {
                "_id" : ObjectId("50883ddf27c133cc4c000018")
        },
        {
                "_id" : ObjectId("507d6bea27c133544400000f")
        },
        {
                "_id" : ObjectId("5080101a27c133c665000003")
        },
        {
                "_id" : ObjectId("507d73fb27c133524400001c")
        },
        {
                "_id" : ObjectId("507d7b1b27c1337c44000018")
        },
        {
                "_id" : ObjectId("507d74fa27c1337c44000006")
        },
        {
                "_id" : ObjectId("507ef62827c1339d4e00002a")
        },
        {
                "_id" : ObjectId("50912bdb5ce2643ad26fee66")
        },
        {
                "_id" : ObjectId("50917fe95ce2643ad27003ea")
        }
]
 
SORRY, Only 800 lines of scrollback buffer.
Here's the bottom part of the second command.
        {
                "_id" : ObjectId("507c15f627c1332f13000001"),
                "schoolID" : "550627000691",
                "uuid" : "abff9707-e188-47f1-b5b9-ec4b23c6bbe7",
                "IP" : "216.56.36.18",
                "atSchool" : "yes",
                "location" : "",
                "onNetwork" : "yes",
                "role" : "technology",
                "email" : "",
                "time" : {
                        "date" : "2012-10-15 13:56:06",
                        "timezone_type" : NumberLong(3),
                        "timezone" : "UTC"
                },
                "downloadSpeed" : "32.36",
                "uploadSpeed" : "10.88",
                "burstSpeedDown" : "28.04",
                "machineTZ" : "America/Chicago",
                "Ping Results" : [
                        "0.07",
                        "0.05",
                        "0.049",
                        "0.06",
                        "0.053"
                ],
                "Up Ping Results" : [
                        "0.077",
                        "0.074",
                        "0.073",
                        "0.115",
                        "0.073"
                ],
                "Test Results" : {
                        "downloadResults" : [
                                {
                                        "downloadSize" : "11483",
                                        "downloadTime" : "0.053",
                                        "error" : "no",
                                        "downloadSpeed" : 1.73
                                },
                                {
                                        "downloadSize" : "164897",
                                        "downloadTime" : "0.149",
                                        "error" : "no",
                                        "downloadSpeed" : 8.85
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.486",
                                        "error" : "no",
                                        "downloadSpeed" : 20.32
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.709",
                                        "error" : "no",
                                        "downloadSpeed" : 21.11
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.76",
                                        "error" : "no",
                                        "downloadSpeed" : 20.5
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.427",
                                        "error" : "no",
                                        "downloadSpeed" : 25.28
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.284",
                                        "error" : "no",
                                        "downloadSpeed" : 28.1
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.158",
                                        "error" : "no",
                                        "downloadSpeed" : 31.15
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.175",
                                        "error" : "no",
                                        "downloadSpeed" : 30.7
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.115",
                                        "error" : "no",
                                        "downloadSpeed" : 32.36
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.142",
                                        "error" : "no",
                                        "downloadSpeed" : 31.59
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.119",
                                        "error" : "no",
                                        "downloadSpeed" : 32.24
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "1.222",
                                        "error" : "no",
                                        "downloadSpeed" : 29.52
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.554",
                                        "error" : "no",
                                        "downloadSpeed" : 14.13
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "3.845",
                                        "error" : "no",
                                        "downloadSpeed" : 9.38
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "4.542",
                                        "error" : "no",
                                        "downloadSpeed" : 7.94
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "5.147",
                                        "error" : "no",
                                        "downloadSpeed" : 7.01
                                }
                        ],
                        "uploadResults" : [
                                {
                                        "uploadSize" : "15311",
                                        "uploadTime" : "0.22",
                                        "error" : "success",
                                        "uploadSpeed" : 0.56
                                },
                                {
                                        "uploadSize" : "219863",
                                        "uploadTime" : "0.444",
                                        "error" : "success",
                                        "uploadSpeed" : 3.96
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.224",
                                        "error" : "success",
                                        "uploadSpeed" : 10.76
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.21",
                                        "error" : "success",
                                        "uploadSpeed" : 10.88
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.234",
                                        "error" : "success",
                                        "uploadSpeed" : 10.67
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.364",
                                        "error" : "success",
                                        "uploadSpeed" : 9.66
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.25",
                                        "error" : "success",
                                        "uploadSpeed" : 5.85
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.265",
                                        "error" : "success",
                                        "uploadSpeed" : 10.41
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.257",
                                        "error" : "success",
                                        "uploadSpeed" : 10.48
                                }
                        ]
                },
                "machine" : {
                        "browser" : "Safari 5.1.5/Mac OS X 10.6.8",
                        "OS" : "Mac OS X 10.6.8",
                        "hasJava" : "yes",
                        "hasFlash" : "yes"
                },
                "CDN" : "commondatastorage.googleapis.com/educationsuperhighway",
                "NewSchool" : {
                        "Name" : "",
                        "Address1" : "",
                        "Address2" : "",
                        "City" : "",
                        "State" : "",
                        "Zip" : ""
                }
        },
        {
                "_id" : ObjectId("507c193e27c1334113000007"),
                "schoolID" : "550627000691",
                "uuid" : "69c6e1a3-c638-4c81-9b28-44c40d427917",
                "IP" : "216.56.36.18",
                "atSchool" : "yes",
                "location" : "",
                "onNetwork" : "yes",
                "role" : "technology",
                "email" : "",
                "time" : {
                        "date" : "2012-10-15 14:10:06",
                        "timezone_type" : NumberLong(3),
                        "timezone" : "UTC"
                },
                "downloadSpeed" : "15.76",
                "uploadSpeed" : "11.04",
                "burstSpeedDown" : "32.29",
                "machineTZ" : "America/Chicago",
                "Ping Results" : [
                        "0.059",
                        "0.059",
                        "0.055",
                        "0.059",
                        "0.05"
                ],
                "Up Ping Results" : [
                        "0.077",
                        "0.074",
                        "0.077",
                        "0.074",
                        "0.079"
                ],
                "Test Results" : {
                        "downloadResults" : [
                                {
                                        "downloadSize" : "11483",
                                        "downloadTime" : "0.056",
                                        "error" : "no",
                                        "downloadSpeed" : 1.64
                                },
                                {
                                        "downloadSize" : "164897",
                                        "downloadTime" : "0.175",
                                        "error" : "no",
                                        "downloadSpeed" : 7.54
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.65",
                                        "error" : "no",
                                        "downloadSpeed" : 15.2
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.289",
                                        "error" : "no",
                                        "downloadSpeed" : 15.76
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.378",
                                        "error" : "no",
                                        "downloadSpeed" : 15.17
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.302",
                                        "error" : "no",
                                        "downloadSpeed" : 15.67
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.574",
                                        "error" : "no",
                                        "downloadSpeed" : 14.02
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.348",
                                        "error" : "no",
                                        "downloadSpeed" : 15.36
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.497",
                                        "error" : "no",
                                        "downloadSpeed" : 14.45
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "2.638",
                                        "error" : "no",
                                        "downloadSpeed" : 13.68
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "3.307",
                                        "error" : "no",
                                        "downloadSpeed" : 10.91
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "4.379",
                                        "error" : "no",
                                        "downloadSpeed" : 8.24
                                },
                                {
                                        "downloadSize" : "4509613",
                                        "downloadTime" : "4.469",
                                        "error" : "no",
                                        "downloadSpeed" : 8.07
                                }
                        ],
                        "uploadResults" : [
                                {
                                        "uploadSize" : "15311",
                                        "uploadTime" : "0.212",
                                        "error" : "success",
                                        "uploadSpeed" : 0.58
                                },
                                {
                                        "uploadSize" : "219863",
                                        "uploadTime" : "0.392",
                                        "error" : "success",
                                        "uploadSpeed" : 4.49
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.28",
                                        "error" : "success",
                                        "uploadSpeed" : 10.29
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.074",
                                        "error" : "success",
                                        "uploadSpeed" : 6.35
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.228",
                                        "error" : "success",
                                        "uploadSpeed" : 10.72
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.202",
                                        "error" : "success",
                                        "uploadSpeed" : 10.96
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "1.193",
                                        "error" : "success",
                                        "uploadSpeed" : 11.04
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.31",
                                        "error" : "success",
                                        "uploadSpeed" : 5.7
                                }
                        ]
                },
                "machine" : {
                        "browser" : "Safari 5.1.5/Mac OS X 10.6.8",
                        "OS" : "Mac OS X 10.6.8",
                        "hasJava" : "yes",
                        "hasFlash" : "yes"
                },
                "CDN" : "commondatastorage.googleapis.com/educationsuperhighway",
                "NewSchool" : {
                        "Name" : "",
                        "Address1" : "",
                        "Address2" : "",
                        "City" : "",
                        "State" : "",
                        "Zip" : ""
                }
        },
        {
                "_id" : ObjectId("507c1b7c27c133331300000f"),
                "schoolID" : "550627000691",
                "uuid" : "be58fa0a-c602-4e04-acc9-14cecc98c4fd",
                "IP" : "216.56.36.18",
                "atSchool" : "yes",
                "location" : "",
                "onNetwork" : "yes",
                "role" : "teacher",
                "email" : "",
                "time" : {
                        "date" : "2012-10-15 14:19:40",
                        "timezone_type" : NumberLong(3),
                        "timezone" : "UTC"
                },
                "downloadSpeed" : "9.71",
                "uploadSpeed" : "4.58",
                "burstSpeedDown" : "8.04",
                "machineTZ" : "America/Chicago",
                "Ping Results" : [
                        "0.057",
                        "0.063",
                        "0.115",
                        "0.055",
                        "0.052"
                ],
                "Up Ping Results" : [
                        "0.074",
                        "0.074",
                        "0.078",
                        "0.076",
                        "0.082"
                ],
                "Test Results" : {
                        "downloadResults" : [
                                {
                                        "downloadSize" : "11483",
                                        "downloadTime" : "0.06",
                                        "error" : "no",
                                        "downloadSpeed" : 1.53
                                },
                                {
                                        "downloadSize" : "164897",
                                        "downloadTime" : "0.18",
                                        "error" : "no",
                                        "downloadSpeed" : 7.33
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "2.057",
                                        "error" : "no",
                                        "downloadSpeed" : 4.8
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.017",
                                        "error" : "no",
                                        "downloadSpeed" : 9.71
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.917",
                                        "error" : "no",
                                        "downloadSpeed" : 5.15
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "2.896",
                                        "error" : "no",
                                        "downloadSpeed" : 3.41
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "2.182",
                                        "error" : "no",
                                        "downloadSpeed" : 4.53
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "2.136",
                                        "error" : "no",
                                        "downloadSpeed" : 4.62
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.963",
                                        "error" : "no",
                                        "downloadSpeed" : 5.03
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.785",
                                        "error" : "no",
                                        "downloadSpeed" : 12.58
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.352",
                                        "error" : "no",
                                        "downloadSpeed" : 7.31
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.628",
                                        "error" : "no",
                                        "downloadSpeed" : 6.07
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "4.916",
                                        "error" : "no",
                                        "downloadSpeed" : 2.01
                                }
                        ],
                        "uploadResults" : [
                                {
                                        "uploadSize" : "15311",
                                        "uploadTime" : "0.194",
                                        "error" : "success",
                                        "uploadSpeed" : 0.63
                                },
                                {
                                        "uploadSize" : "219863",
                                        "uploadTime" : "0.379",
                                        "error" : "success",
                                        "uploadSpeed" : 4.64
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "3.469",
                                        "error" : "success",
                                        "uploadSpeed" : 3.8
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.873",
                                        "error" : "success",
                                        "uploadSpeed" : 4.58
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "3.744",
                                        "error" : "success",
                                        "uploadSpeed" : 3.52
                                }
                        ]
                },
                "machine" : {
                        "browser" : "Firefox 10.0.1/Mac OS X 10.6",
                        "OS" : "Mac OS X 10.6",
                        "hasJava" : "yes",
                        "hasFlash" : "yes"
                },
                "CDN" : "commondatastorage.googleapis.com/educationsuperhighway",
                "NewSchool" : {
                        "Name" : "",
                        "Address1" : "",
                        "Address2" : "",
                        "City" : "",
                        "State" : "",
                        "Zip" : ""
                }
        },
        {
                "_id" : ObjectId("507c1ba127c1339e13000004"),
                "schoolID" : "550627000691",
                "uuid" : "be3ade7a-83b6-484a-a99b-d971ad3b8172",
                "IP" : "216.56.36.18",
                "atSchool" : "yes",
                "location" : "",
                "onNetwork" : "yes",
                "role" : "teacher",
                "email" : "",
                "time" : {
                        "date" : "2012-10-15 14:20:17",
                        "timezone_type" : NumberLong(3),
                        "timezone" : "UTC"
                },
                "downloadSpeed" : "33.48",
                "uploadSpeed" : "6.55",
                "burstSpeedDown" : "20.44",
                "machineTZ" : "America/Chicago",
                "Ping Results" : [
                        "0.093",
                        "0.057",
                        "0.058",
                        "0.054",
                        "0.053"
                ],
                "Up Ping Results" : [
                        "0.075",
                        "0.076",
                        "0.074",
                        "0.074",
                        "0.075"
                ],
                "Test Results" : {
                        "downloadResults" : [
                                {
                                        "downloadSize" : "11483",
                                        "downloadTime" : "0.065",
                                        "error" : "no",
                                        "downloadSpeed" : 1.41
                                },
                                {
                                        "downloadSize" : "164897",
                                        "downloadTime" : "0.112",
                                        "error" : "no",
                                        "downloadSpeed" : 11.78
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.48",
                                        "error" : "no",
                                        "downloadSpeed" : 6.67
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.663",
                                        "error" : "no",
                                        "downloadSpeed" : 5.94
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.729",
                                        "error" : "no",
                                        "downloadSpeed" : 13.55
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.3",
                                        "error" : "no",
                                        "downloadSpeed" : 32.92
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "2.999",
                                        "error" : "no",
                                        "downloadSpeed" : 3.29
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.723",
                                        "error" : "no",
                                        "downloadSpeed" : 13.66
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.295",
                                        "error" : "no",
                                        "downloadSpeed" : 33.48
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.432",
                                        "error" : "no",
                                        "downloadSpeed" : 22.86
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.77",
                                        "error" : "no",
                                        "downloadSpeed" : 5.58
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.953",
                                        "error" : "no",
                                        "downloadSpeed" : 10.36
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.625",
                                        "error" : "no",
                                        "downloadSpeed" : 15.8
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "0.625",
                                        "error" : "no",
                                        "downloadSpeed" : 15.8
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.329",
                                        "error" : "no",
                                        "downloadSpeed" : 7.43
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.355",
                                        "error" : "no",
                                        "downloadSpeed" : 7.29
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.508",
                                        "error" : "no",
                                        "downloadSpeed" : 6.55
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.64",
                                        "error" : "no",
                                        "downloadSpeed" : 6.02
                                },
                                {
                                        "downloadSize" : "1234664",
                                        "downloadTime" : "1.933",
                                        "error" : "no",
                                        "downloadSpeed" : 5.11
                                }
                        ],
                        "uploadResults" : [
                                {
                                        "uploadSize" : "15311",
                                        "uploadTime" : "0.177",
                                        "error" : "success",
                                        "uploadSpeed" : 0.69
                                },
                                {
                                        "uploadSize" : "219863",
                                        "uploadTime" : "0.537",
                                        "error" : "success",
                                        "uploadSpeed" : 3.28
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.561",
                                        "error" : "success",
                                        "uploadSpeed" : 5.14
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.729",
                                        "error" : "success",
                                        "uploadSpeed" : 4.83
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.01",
                                        "error" : "success",
                                        "uploadSpeed" : 6.55
                                },
                                {
                                        "uploadSize" : "1646219",
                                        "uploadTime" : "2.944",
                                        "error" : "success",
                                        "uploadSpeed" : 4.47
                                }
                        ]
                },
                "machine" : {
                        "browser" : "Firefox 10.0.1/Mac OS X 10.6",
                        "OS" : "Mac OS X 10.6",
                        "hasJava" : "yes",
                        "hasFlash" : "yes"
                },
                "CDN" : "commondatastorage.googleapis.com/educationsuperhighway",
                "NewSchool" : {
                        "Name" : "",
                        "Address1" : "",
                        "Address2" : "",
                        "City" : "",
                        "State" : "",
                        "Zip" : ""
                }
        }
]
>

Comment by Aaron Staple [ 13/Nov/12 ]

Thanks, ok now I'd like to try and find a few docs that are not being returned in the first one. Could you run this:

ids = db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }, { _id:1 } ).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).toArray();
 
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691", $nor:ids }).hint({$natural:1}).limit( -5 ).toArray();

Comment by William Miskovetz [ 13/Nov/12 ]

 
MongoDB shell version: 2.2.1
connecting to: EducationSuperhighway
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).itcount()
113
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({$natural:1}).itcount()
 
5641
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).itcount()
5641
>
 

Comment by Aaron Staple [ 13/Nov/12 ]

Hi William,

Sorry could you also send:

db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).itcount()
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({$natural:1}).itcount()
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).itcount()

Comment by William Miskovetz [ 13/Nov/12 ]

Sorry for the cropped lines. Terminal not set to wrap long lines.

Comment by William Miskovetz [ 13/Nov/12 ]

 
MongoDB shell version: 2.2.1
connecting to: EducationSuperhighway
>
> db.Results.getIndexes()
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "ns" : "EducationSuperhighway.Results",
                "name" : "_id_"
        },
        {
                "v" : 1,
                "key" : {
                        "uuid" : 1
                },
                "ns" : "EducationSuperhighway.Results",
                "name" : "uuid_1"
        },
        {
                "v" : 1,
                "key" : {
                        "Test Results" : 1,
                        "atSchool" : 1,
                        "onNetwork" : 1,
                        "schoolID" : 1
                },
                "ns" : "EducationSuperhighway.Results",
                "name" : "Test Results_1_atSchool_1_onNetwork_1_schoolID_1"
        }
]
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).explain( true )
{
        "cursor" : "BtreeCursor Test Results_1_atSchool_1_onNetwork_1_schoolID_1",
        "isMultiKey" : false,
        "n" : 113,
        "nscannedObjects" : 177,
        "nscanned" : 6531,
        "nscannedObjectsAllPlans" : 177,
        "nscannedAllPlans" : 6531,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 50,
        "indexBounds" : {
                "Test Results" : [
                        [
                                {
                                        "$minElement" : 1
                                },
                                {
                                        "$maxElement" : 1
                                }
                        ]
                ],
                "atSchool" : [
                        [
                                "yes",
                                "yes"
                        ]
                ],
                "onNetwork" : [
                        [
                                "yes",
                                "yes"
                        ]
                ],
                "schoolID" : [
                        [
                                "550627000691",
                                "550627000691"
                        ]
                ]
        },
        "allPlans" : [
                {
                        "cursor" : "BtreeCursor Test Results_1_atSchool_1_onNetwork_1_schoolID_1",
                        "n" : 113,
                        "nscannedObjects" : 177,
                        "nscanned" : 6531,
                        "indexBounds" : {
                                "Test Results" : [
                                        [
                                                {
                                                        "$minElement" : 1
                                                },
                                                {
                                                        "$maxElement" : 1
                                                }
                                        ]
                                ],
                                "atSchool" : [
                                        [
                                                "yes",
                                                "yes"
                                        ]
                                ],
                                "onNetwork" : [
                                        [
                                                "yes",
                                                "yes"
                                        ]
                                ],
                                "schoolID" : [
                                        [
                                                "550627000691",
                                                "550627000691"
                                        ]
                                ]
                        }
                }
        ],
        "server" : "ip-10-100-241-138:27017"
}
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).count()
5641
> db.Results.count( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" })
5641
> db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).explain( true )
{
        "cursor" : "BasicCursor",
        "isMultiKey" : false,
        "n" : 5641,
        "nscannedObjects" : 114641,
        "nscanned" : 114641,
        "nscannedObjectsAllPlans" : 114641,
        "nscannedAllPlans" : 114641,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 1,
        "nChunkSkips" : 0,
        "millis" : 568,
        "indexBounds" : {
 
        },
        "allPlans" : [
                {
                        "cursor" : "BasicCursor",
                        "n" : 5641,
                        "nscannedObjects" : 114641,
                        "nscanned" : 114641,
                        "indexBounds" : {
 
                        }
                }
        ],
        "server" : "ip-10-100-241-138:27017"
}
>
 

Comment by Aaron Staple [ 13/Nov/12 ]

Hi William,

Just to get some more info, can you please run the following and send the full commands you enter and the full output for each command:

db.Results.getIndexes()
 
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).explain( true )
 
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).hint({"Test Results" : 1, "atSchool" : 1, "onNetwork" : 1, "schoolID" : 1}).count()
 
db.Results.count( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" })
 
db.Results.find( { "Test Results" : { "$exists" : true }, "atSchool" : "yes", "onNetwork" : "yes", "schoolID" : "550627000691" }).explain( true )

Comment by William Miskovetz [ 13/Nov/12 ]

SERVER-2677 is the other bug I'm referring to.

Generated at Thu Feb 08 03:15:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.