[SERVER-14706] Queries that use negated $type predicate over a field may return incomplete results when an index is present on that field Created: 28/Jul/14  Updated: 11/Mar/15  Resolved: 31/Jul/14

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 2.6.1, 2.6.3
Fix Version/s: 2.6.5, 2.7.5

Type: Bug Priority: Major - P3
Reporter: Stennie Steneker (Inactive) Assignee: David Storch
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
Tested
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Insert a document into a new collection, it has a single field x which is a 32-bit int:

    > db.typetest.insert({x:NumberInt(1)})
    WriteResult({ "nInserted" : 1 })
    > db.typetest.find()
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

Note that $type only matches type 16 (32-bit int), as expected:

    > db.typetest.find({x:{$type:1}})
    > db.typetest.find({x:{$type:16}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
    > db.typetest.find({x:{$type:17}})
    > db.typetest.find({x:{$type:18}})

I need to search for documents that do not match the given type, and $not works as expected:

    > db.typetest.find({x:{$not:{$type:1}}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
    > db.typetest.find({x:{$not:{$type:2}}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
    > db.typetest.find({x:{$not:{$type:16}}})
    > db.typetest.find({x:{$not:{$type:18}}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

After creating an index, $type works but $not $type does not differentiate between the 3 numeric types:

    > db.typetest.ensureIndex({x:1})
    {
    	"createdCollectionAutomatically" : false,
    	"numIndexesBefore" : 1,
    	"numIndexesAfter" : 2,
    	"ok" : 1
    }
    > 
    > db.typetest.find({x:{$type:1}})
    > db.typetest.find({x:{$type:16}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
    > db.typetest.find({x:{$type:17}})
    > db.typetest.find({x:{$type:18}})
    > 
    > db.typetest.find({x:{$not:{$type:1}}})
    > db.typetest.find({x:{$not:{$type:16}}})
    > db.typetest.find({x:{$not:{$type:18}}})
    > db.typetest.find({x:{$not:{$type:2}}})
    { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

Participants:

 Description   
Issue Status as of Aug 22, 2014

ISSUE SUMMARY
Queries that use negated $type predicate over a given field may not produce the expected results when there's an index present on that field.

For example, the BSON type of an integer is 16. The following sequence displays correct behavior:

> db.foo.drop()
> db.foo.insert({n : 1, data : NumberInt(1)})
> db.foo.insert({n : 2, data : NumberLong(2)})
> db.foo.insert({n : 3, data : NumberInt(3)})
> db.foo.insert({n : 4, data : NumberLong(4)})
> db.foo.insert({n : 5, data : NumberInt(4)})
> db.foo.insert({n : 6, data : NumberLong(6)})
> db.foo.find({data:{$not:{$type:16}}}, {_id : 0}) // CORRECT
{ "n" : 2, "data" : NumberLong(2) }
{ "n" : 4, "data" : NumberLong(4) }
{ "n" : 6, "data" : NumberLong(6) }

After creating an index on field "data", queries that use $type on that field work as expected, but queries that use negated $type on "data" may return an incorrect result set:

> db.foo.ensureIndex({data : 1})
> db.foo.find({data:{$not:{$type:16}}}, {_id : 0}) // INCORRECT - does not return any document
>
> db.foo.find({data:{$not:{$type:2}}}, {_id : 0}) // CORRECT
{ "n" : 1, "data" : 1 }
{ "n" : 2, "data" : NumberLong(2) }
{ "n" : 3, "data" : 3 }
{ "n" : 4, "data" : NumberLong(4) }
{ "n" : 5, "data" : 4 }
{ "n" : 6, "data" : NumberLong(6) }

USER IMPACT
Users may receive incomplete result sets. This also applies to queries that are part of an update operation.

WORKAROUNDS
If documents have another indexed field, and queries do not include negated $type clauses over that field, it is possible to hint on this index as a workaround:

> db.foo.ensureIndex({n : 1})
> db.foo.find({data:{$not:{$type:16}}, n : {$gt : 1}}, {_id : 0}) // INCORRECT - does not return any document
>
> db.foo.find({data:{$not:{$type:16}}, n : {$gt : 1}}, {_id : 0}).hint({n:1}) // CORRECT
{ "n" : 2, "data" : NumberLong(2) }
{ "n" : 4, "data" : NumberLong(4) }
{ "n" : 6, "data" : NumberLong(6) }

AFFECTED VERSIONS
MongoDB 2.6 production releases up to 2.6.4 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.5 production release.

RESOLUTION DETAILS
Prevent negated $type predicates from using an index.

Original description

Originally reported on StackOverflow.

I'm trying to write an automated test to ensure that some fields in my collection have a consistent type across documents. For each of these fields, I have an expected type. So I'm querying for documents that have that field with a different type, ignoring documents where that field is missing or null.

When an index exists on the field, it seems that numeric types 1 (double), 16 (32-bit int) and 18 (64-bit int) are not differentiated. I'm using mongo 2.6.1.



 Comments   
Comment by Githook User [ 10/Sep/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14706 SERVER-14706 negated $type predicates should not use an index

(cherry picked from commit 6bfddeb720b51b88533c7ae453dc5c702fdc58a5)
Branch: v2.6
https://github.com/mongodb/mongo/commit/976747c1224362120f8ee5079f542cb4f5082d32

Comment by Githook User [ 31/Jul/14 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-14706 negated $type predicates should not use an index
Branch: master
https://github.com/mongodb/mongo/commit/6bfddeb720b51b88533c7ae453dc5c702fdc58a5

Generated at Thu Feb 08 03:35:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.