Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14706

Queries that use negated $type predicate over a field may return incomplete results when an index is present on that field

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 2.6.1, 2.6.3
    • Fix Version/s: 2.6.5, 2.7.5
    • Component/s: Querying
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      Insert a document into a new collection, it has a single field x which is a 32-bit int:

          > db.typetest.insert({x:NumberInt(1)})
          WriteResult({ "nInserted" : 1 })
          > db.typetest.find()
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

      Note that $type only matches type 16 (32-bit int), as expected:

          > db.typetest.find({x:{$type:1}})
          > db.typetest.find({x:{$type:16}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
          > db.typetest.find({x:{$type:17}})
          > db.typetest.find({x:{$type:18}})

      I need to search for documents that do not match the given type, and $not works as expected:

          > db.typetest.find({x:{$not:{$type:1}}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
          > db.typetest.find({x:{$not:{$type:2}}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
          > db.typetest.find({x:{$not:{$type:16}}})
          > db.typetest.find({x:{$not:{$type:18}}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

      After creating an index, $type works but $not $type does not differentiate between the 3 numeric types:

          > db.typetest.ensureIndex({x:1})
          {
          	"createdCollectionAutomatically" : false,
          	"numIndexesBefore" : 1,
          	"numIndexesAfter" : 2,
          	"ok" : 1
          }
          > 
          > db.typetest.find({x:{$type:1}})
          > db.typetest.find({x:{$type:16}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }
          > db.typetest.find({x:{$type:17}})
          > db.typetest.find({x:{$type:18}})
          > 
          > db.typetest.find({x:{$not:{$type:1}}})
          > db.typetest.find({x:{$not:{$type:16}}})
          > db.typetest.find({x:{$not:{$type:18}}})
          > db.typetest.find({x:{$not:{$type:2}}})
          { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

      Show
      Insert a document into a new collection, it has a single field x which is a 32-bit int: > db.typetest.insert({x:NumberInt(1)}) WriteResult({ "nInserted" : 1 }) > db.typetest.find() { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } Note that $type only matches type 16 (32-bit int), as expected: > db.typetest.find({x:{$type:1}}) > db.typetest.find({x:{$type:16}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } > db.typetest.find({x:{$type:17}}) > db.typetest.find({x:{$type:18}}) I need to search for documents that do not match the given type, and $not works as expected: > db.typetest.find({x:{$not:{$type:1}}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } > db.typetest.find({x:{$not:{$type:2}}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } > db.typetest.find({x:{$not:{$type:16}}}) > db.typetest.find({x:{$not:{$type:18}}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } After creating an index, $type works but $not $type does not differentiate between the 3 numeric types: > db.typetest.ensureIndex({x:1}) { "createdCollectionAutomatically" : false, "numIndexesBefore" : 1, "numIndexesAfter" : 2, "ok" : 1 } > > db.typetest.find({x:{$type:1}}) > db.typetest.find({x:{$type:16}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 } > db.typetest.find({x:{$type:17}}) > db.typetest.find({x:{$type:18}}) > > db.typetest.find({x:{$not:{$type:1}}}) > db.typetest.find({x:{$not:{$type:16}}}) > db.typetest.find({x:{$not:{$type:18}}}) > db.typetest.find({x:{$not:{$type:2}}}) { "_id" : ObjectId("53d52620eb3748b3f9cde293"), "x" : 1 }

      Description

      Issue Status as of Aug 22, 2014

      ISSUE SUMMARY
      Queries that use negated $type predicate over a given field may not produce the expected results when there's an index present on that field.

      For example, the BSON type of an integer is 16. The following sequence displays correct behavior:

      > db.foo.drop()
      > db.foo.insert({n : 1, data : NumberInt(1)})
      > db.foo.insert({n : 2, data : NumberLong(2)})
      > db.foo.insert({n : 3, data : NumberInt(3)})
      > db.foo.insert({n : 4, data : NumberLong(4)})
      > db.foo.insert({n : 5, data : NumberInt(4)})
      > db.foo.insert({n : 6, data : NumberLong(6)})
      > db.foo.find({data:{$not:{$type:16}}}, {_id : 0}) // CORRECT
      { "n" : 2, "data" : NumberLong(2) }
      { "n" : 4, "data" : NumberLong(4) }
      { "n" : 6, "data" : NumberLong(6) }

      After creating an index on field "data", queries that use $type on that field work as expected, but queries that use negated $type on "data" may return an incorrect result set:

      > db.foo.ensureIndex({data : 1})
      > db.foo.find({data:{$not:{$type:16}}}, {_id : 0}) // INCORRECT - does not return any document
      >
      > db.foo.find({data:{$not:{$type:2}}}, {_id : 0}) // CORRECT
      { "n" : 1, "data" : 1 }
      { "n" : 2, "data" : NumberLong(2) }
      { "n" : 3, "data" : 3 }
      { "n" : 4, "data" : NumberLong(4) }
      { "n" : 5, "data" : 4 }
      { "n" : 6, "data" : NumberLong(6) }

      USER IMPACT
      Users may receive incomplete result sets. This also applies to queries that are part of an update operation.

      WORKAROUNDS
      If documents have another indexed field, and queries do not include negated $type clauses over that field, it is possible to hint on this index as a workaround:

      > db.foo.ensureIndex({n : 1})
      > db.foo.find({data:{$not:{$type:16}}, n : {$gt : 1}}, {_id : 0}) // INCORRECT - does not return any document
      >
      > db.foo.find({data:{$not:{$type:16}}, n : {$gt : 1}}, {_id : 0}).hint({n:1}) // CORRECT
      { "n" : 2, "data" : NumberLong(2) }
      { "n" : 4, "data" : NumberLong(4) }
      { "n" : 6, "data" : NumberLong(6) }

      AFFECTED VERSIONS
      MongoDB 2.6 production releases up to 2.6.4 are affected by this issue.

      FIX VERSION
      The fix is included in the 2.6.5 production release.

      RESOLUTION DETAILS
      Prevent negated $type predicates from using an index.

      Original description

      Originally reported on StackOverflow.

      I'm trying to write an automated test to ensure that some fields in my collection have a consistent type across documents. For each of these fields, I have an expected type. So I'm querying for documents that have that field with a different type, ignoring documents where that field is missing or null.

      When an index exists on the field, it seems that numeric types 1 (double), 16 (32-bit int) and 18 (64-bit int) are not differentiated. I'm using mongo 2.6.1.

        Attachments

          Activity

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: