[SERVER-1735] Inconsistent behaviour for $exists: false and $type: 10 in query Created: 04/Sep/10  Updated: 12/Jul/16  Resolved: 03/Jun/11

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 1.6.1, 1.6.2, 1.7.0
Fix Version/s: 1.9.1

Type: Bug Priority: Major - P3
Reporter: Andrea Spacca Assignee: Aaron Staple
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tested as bogus in macosx 10.6.4 server 64bit and Ubuntu 9.10 64bit


Issue Links:
Depends
depends on SERVER-1587 $exists:false not working correctly Closed
Operating System: ALL
Participants:

 Description   

key: {$exists: false} / key: {$type: 10} / key: null quering gives inconsistent and unpredictable behaviour, so to workraound needed to implement querying of this kind of parameters

shell script example:
db.testcollection.ensureIndex(

{date: -1, country_code: 1, user_id: 1}

,

{unique: 1, background: 1}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), tot_visit: 100}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), country_code: "IT", tot_visit: 77}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), country_code: "ES", tot_visit: 23}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), country_code: "ES", user_id: "and...@spacca.org", tot_visit: 11}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), country_code: "ES", user_id: "andrea.spa...@gmail.com", tot_visit: 5}

);
db.testcollection.insert(

{ date: new Date("08/27/2010"), country_code: "ES", user_id: "andrea.spa...@progloedizioni.com", tot_visit: 7}

);

db.testcollection.find(

{date: new Date("08/27/2010")}

).count(); // 6 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}}).count(); // 5 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: false}}).count(); // 1 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$type: 10}}).count(); // 1 [OK]
db.testcollection.find(

{date: new Date("08/27/2010"), country_code: null}

).count(); // 1 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}, user_id: {$exists: true}}).count(); // 3 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}, user_id: {$exists: false}}).count(); // 2 [OK]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}, user_id: {$type: 10}}).count(); // 0 [??]
db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}, user_id: null}).count(); // 2 [OK]

changing the order of key querying or indexing produces slightly different results, always bogus



 Comments   
Comment by auto [ 03/Jun/11 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-1735 add a real field value test case
Branch: master
https://github.com/mongodb/mongo/commit/ba9daf00db064ef9a284f5b72bb3e43a0ca141bb

Comment by Aaron Staple [ 03/Jun/11 ]

Our behavior now is

{a:null}

- match if a is null or missing
{a:{$exists:true}} - match if a is present
{a:{$exists:false}} - match if a is missing
{a:{$type:10}} - match if a is null, but not if a is missing or present and nonnull

Comment by auto [ 02/Jun/11 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-1735 type:10 query matches null value but not missing value, hence an index check is insufficient to do a match
Branch: master
https://github.com/mongodb/mongo/commit/37d36b7cf1f3a0b8b2a7639b91976673377fd47c

Comment by Aaron Staple [ 05/Apr/11 ]

The exists:false failure is SERVER-1587

Comment by Guanqun Lu [ 28/Jan/11 ]

come up with a more compact fix, as this type info is already maintained in bm.

https://github.com/mongodb/mongo/pull/14/

any comments?

Comment by Guanqun Lu [ 13/Jan/11 ]

I did some hacking on this problem and found

db/matcher.cpp bool Matcher::matches(const BSONObj& jsobj , MatchDetails * details )

764 if ( cmp == 0 ) {
765 /* missing is ok iff we were looking for null */
766 if ( m.type() == jstNULL || m.type() == Undefined || ( bm.compareOp == BSONObj::opIN && bm.myset->count( staticNull.firstElement() ) > 0 ) ) {
767 if ( ( bm.compareOp == BSONObj::NE ) ^ bm.isNot )

{ 768 return false; 769 }

at line 766 shown above, for ElementMatcher

{ $type : 10 }

to match user_id, at this point, m.type() is NumberDouble (value is 10.0), and the compareOp is 15 (opTYPE). we should take this case into consideration because we're actually looking for null.

the corresponding fix is here. and i've added a jstest for this bug.
https://github.com/guanqun/mongo/commit/07201282d7b91323280ae9d0465ba8872208829c

please review,
this patch changes the core part of the server, hope i'm not missing something and causes regression by accident.

Comment by Doug Mayer [ 03/Nov/10 ]

The following line is noted as OK above:

db.testcollection.find({date: new Date("08/27/2010"), country_code: {$exists: true}, user_id: {$exists: false}}).count(); // 2 [OK]

On Mac OSX 64-bit, v1.6.3, this line actually returns ZERO records. There seems to be a lot of inconsistent behavior around $exists in 1.6.

Generated at Thu Feb 08 02:57:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.