[SERVER-1475] {field: {$type:"array"}} should return documents where "field" is an array Created: 24/Jul/10  Updated: 04/Nov/20  Resolved: 08/May/17

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 1.4.4
Fix Version/s: 3.5.7

Type: Bug Priority: Major - P3
Reporter: Tatyana Tvardovskaya Assignee: David Storch
Resolution: Done Votes: 14
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 10.04(lucid), x64


Issue Links:
Documented
is documented by DOCS-10226 New behavior for {field: {$type: "arr... Closed
Duplicate
is duplicated by SERVER-1854 $type not working properly with arrays Closed
is duplicated by SERVER-38770 "$type" cannot match array in version... Closed
Related
related to SERVER-51303 Lookup stage followed by $match on ty... Closed
related to SERVER-29163 Version match expressions Backlog
Backwards Compatibility: Minor Change
Operating System: ALL
Sprint: Query 2017-05-08, Query 2017-05-29
Participants:

 Description   

Operator $type does not finds array (type=4) elements.

It tested db, 'aList' field is array. There are records with empty arrays and not empty.
But request to count elemets by type = array brings zero (0) results;
>db.mydata.count({ aList: {$type: 4 } }); //type = Array
0

Meantime, the following check shows that there are arrays, and som of them are non-empty:
> var obj = db.mydata.count({ aList: {$type:3} }); //type = Object
> obj
33276

> var zero = db.mydata.count({ aList: {$size: 0 } }); //empty arrays
> zero
421251
> var all = db.mydata.count({ }); //all records
> all
454527
> all - zero // number of non- empty arrays
33276

As you see, obj == (all - zero) - all arrays are treated as Objects(type=3)!
Expected: array fileds should be treated as Arrays (type=4).



 Comments   
Comment by David Storch [ 12/May/17 ]

This change will affect users on 3.4.x who have partial indexes whose partialFilterExpression has a $type:"array" expression and wish to upgrade to 3.6. As part of the upgrade process, such users must rebuild any such partial index when first starting a 3.6 node. Failing to do so could cause the user to experience a mix of the old and new $type:"array" semantics. We should clearly document this wrinkle in the 3.6 compatibility notes so that users who may be affected can take the appropriate action.

Note that this also applies to $type:4, since 4 is the type code for BSON arrays.

Comment by Githook User [ 08/May/17 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-1475 Make $type:'array' match outer arrays.

Prior to this change, the semantics of $type were that it
only matches documents with nested arrays, e.g. {x: {$type:
'array'}} would match

{x: [1, [2, 3]]}

but not

{x: [1, 2, 3]}

. This is inconsistent with the matching semantics for
other query predicate operators, which apply both to the
individual array elements and the array as a whole.

The new behavior is that single-level arrays as well as
nested arrays will match {$type:'array'}. This is a breaking
change for applications which rely on the old behavior.
Branch: master
https://github.com/mongodb/mongo/commit/c6d0c0dd01c328d8ed1a5744485b29826b1fb2c9

Comment by Asya Kamsky [ 02/Feb/17 ]

As of 3.4 the proposed workaround works fine on two dimensional arrays too:

db.test1.find({$or: [{"a.0": {$size: 0}}, {"a.0": {$elemMatch: {$exists: true}}}]})
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef1"), "a" : [ [ 10, 11 ], [ 20, 21 ] ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef2"), "a" : [ [ 1 ], [ 2 ], [ 3 ] ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef3"), "a" : [ [ ] ] }
Fetched 3 record(s) in 2ms

Comment by aurelien lambert [ 06/Mar/14 ]

The javascript workaround is as said not efficient, and then non javascript workaround requires an $or at the root, which makes it tricky if one needs to check if several fields are arrays.

Wouldn't it be more simple if the $all:[] could do it ? It would look like

> db.test1.find()
{ "a" : [ 0 ] }
{ "a" : [ ] }
{ "a" : 0 }
> db.test1.find({a : {$all : []}})
{ "a" : [ 0 ] }
{ "a" : [ ] }

Isn't it simple ?

Comment by Thomas Zahn [ 14/Feb/14 ]

That's a nice work-around, however it doesn't work for nested arrays.
Consider these arrays:

mongos> db.test1.find()
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebeef"), "a" : [  7,  10 ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef0"), "a" : [  1,  2,  3 ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef1"), "a" : [  [  10,  11 ],  [  20,  21 ] ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef2"), "a" : [  [  1 ],  [  2 ],  [  3 ] ] }
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef3"), "a" : [  [ ] ] }

Suppose one now wanted to explicitly find those documents where a specific element of a (e.g. a.0) is again an array, i.e. a 2-dimensional array. Unfortunately, only $size seems to work on a.0, whereas $elemMatch fails:

mongos> db.test1.find({$or: [{"a.0": {$size: 0}}, {"a.0": {$elemMatch: {$exists: true}}}]})
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef3"), "a" : [  [ ] ] }
mongos> db.test1.find({"a.0": {$size: 0}})
{ "_id" : ObjectId("52fcf0e3f067bdb1472ebef3"), "a" : [  [ ] ] }
mongos> db.test1.find({"a.0": {$elemMatch: {$exists: true}}})
mongos>

This seems related to the following reported bug in $elemMatch: SERVER-1264.
So, yes, a new $isArray operator would be great.

Comment by Tobia Conforto [ 02/Oct/13 ]

Here is an updated workaround to deal with the edge case of a document with key "0":

> db.test1.find()
{ "_id" : ObjectId("4f88538ef014d20bea9cbb60") }
{ "_id" : ObjectId("4f885397f014d20bea9cbb61"), "a" : 1 }
{ "_id" : ObjectId("4f88539af014d20bea9cbb62"), "a" : [ ] }
{ "_id" : ObjectId("4f88539df014d20bea9cbb63"), "a" : [ 1 ] }
{ "_id" : ObjectId("4f88539ff014d20bea9cbb64"), "a" : [ 1, 2 ] }
{ "_id" : ObjectId("4f885397f014d20bea9cbb65"), "a" : { "0" : "uh" } }
>
> db.test1.find({$or: [{"a": {$size: 0}}, {"a": {$elemMatch: {$exists: true}}}]})
{ "_id" : ObjectId("4f88539af014d20bea9cbb62"), "a" : [ ] }
{ "_id" : ObjectId("4f88539df014d20bea9cbb63"), "a" : [  1 ] }
{ "_id" : ObjectId("4f88539ff014d20bea9cbb64"), "a" : [  1,  2 ] }
>

It tests for a non-empty array using $elemMatch instead of .0
It's pretty ugly though. A new $isArray operator is probably the best solution.

Comment by A. Jesse Jiryu Davis [ 05/Jun/12 ]

What about adding an $isArray operator to deal with the specific use-case of determining if a field is of array type, while leaving the $type operator's current behavior--matching elements of the array rather than the array itself, consistent with other operators?

Comment by Daniel Gottlieb (Inactive) [ 13/Apr/12 ]

As a caveat, doing an existence test for "a.0" will also match a document such as: {"_id": ..., "a": {"0": "foo", "name": "bar"}}, but I'll venture it's an unlikely edge case.

test:PRIMARY> db.test.find({$or: [

{x: []}

, {'x.0': {$exists: true}}]})

{ "_id" : ObjectId("4f885a1a3b5c50b21c2f5311"), "x" : [ ] } { "_id" : ObjectId("4f885a213b5c50b21c2f5312"), "x" : [ "1" ] }

{ "_id" : ObjectId("4f885b923b5c50b21c2f5314"), "x" :

{ "0" : "foo", "name" : "bar" }

}

Comment by Robert Stam [ 13/Apr/12 ]

Here's a workaround that doesn't involve Javascript:

> db.test.find()
{ "_id" : ObjectId("4f88538ef014d20bea9cbb60") }
{ "_id" : ObjectId("4f885397f014d20bea9cbb61"), "a" : 1 }
{ "_id" : ObjectId("4f88539af014d20bea9cbb62"), "a" : [ ] }
{ "_id" : ObjectId("4f88539df014d20bea9cbb63"), "a" : [ 1 ] }
{ "_id" : ObjectId("4f88539ff014d20bea9cbb64"), "a" : [ 1, 2 ] }
>
> db.test.find({"a.0":{$exists:true}})
{ "_id" : ObjectId("4f88539df014d20bea9cbb63"), "a" : [ 1 ] }
{ "_id" : ObjectId("4f88539ff014d20bea9cbb64"), "a" : [ 1, 2 ] }
>
> db.test.find({a:{$size:0}})
{ "_id" : ObjectId("4f88539af014d20bea9cbb62"), "a" : [ ] }
>
> db.test.find({$or:[{a:{$size:0}},{"a.0":{$exists:true}}]})
{ "_id" : ObjectId("4f88539af014d20bea9cbb62"), "a" : [ ] }
{ "_id" : ObjectId("4f88539df014d20bea9cbb63"), "a" : [ 1 ] }
{ "_id" : ObjectId("4f88539ff014d20bea9cbb64"), "a" : [ 1, 2 ] }
>

The existence test for "a.0" doesn't detect empty arrays, but if you need to find empty arrays also you can combine it with a test for {$size:0}.

Comment by Scott Hernandez (Inactive) [ 13/Apr/12 ]

You can use javascript to do the test but that is not very efficient:

> db.arrays.find()
{ "_id" : 1, "a" : [ 1, 2, 3 ] }
{ "_id" : 2, "a" : 1 }
> db.arrays.find("Array.isArray(this.a)")
{ "_id" : 1, "a" : [ 1, 2, 3 ] }

Comment by Luc Dobler [ 13/Feb/12 ]

Does anyone have a workaround to this ?

Comment by Ara T. Howard [ 05/Feb/12 ]

i would say there are four things that make this a bug:

1) the docs clearly state this is the expected behavior http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-%24type

2) without such a feature the query 'where the field "foo" is an array' is impossible, which would be a shame in a schema free document oriented database...

3) searches for $type==3 (object) return arrays, which is extremely counter intuitive

4) searchers for $type==anthing (int, string, etc) return not only where the field has that type, but also where there might exist that type wrapped by an array.

Comment by Alex Litvinok [ 02/Dec/11 ]

This problem in PHP driver is expressed:

$mongo = new Mongo( 'mongodb://localhost' );
$collection = $mongo -> selectDB( 'temp' ) -> selectCollection('A');
 
$count = array();
$collection -> remove();
$collection -> insert(array( 'field' => array( 1, 2, 3 ) ));
 
$countAll = $collection -> find() -> count();
$countArr = $collection -> find( array( 'field' => array( '$type' => 4)) ) -> count();
 
echo sprintf("Total:\t%d%s",  $countAll, PHP_EOL );
echo sprintf("Array:\t%d%s", $countArr , PHP_EOL );

Returned as:

Total:	1
Array:	0

Also, if embedded array:

$mongo = new Mongo( 'mongodb://localhost' );
$collection = $mongo -> selectDB( 'temp' ) -> selectCollection('A');
 
$count = array();
$collection -> remove();
$collection -> insert(array( 'field' => array( 1, 2, 3 ) ));
$collection -> insert(array( 'field' => array( array(1,2,3), 4, 5 ) ));
 
$countAll = $collection -> find() -> count();
$countArr = $collection -> find( array( 'field' => array( '$type' => 4)) ) -> count();
 
echo sprintf("Total:\t%d%s",  $countAll, PHP_EOL );
echo sprintf("Array:\t%d%s", $countArr , PHP_EOL );

Returned as:

Total:	2
Array:	1

Comment by Uladzimir Mihura [ 30/Nov/11 ]

>db.tmp.insert(

{"title":"The Godfather","keywords":["test","word","cool"]}

)
>db.tmp.insert(

{"title":"Fight club","keywords":[["arr1","arr2"],"word","cool"]}

)
>db.tmp.find({"keywords":{"$type":4}})

{ "_id" : ObjectId("4ed61b9c0f0b2248459a6abb"), "title" : "Fight club", "keywords" : [ [ "arr1", "arr2" ], "word", "cool" ] }

Actually it is looking inside the array and not at the field itself.
Is is possible to determine if the field is array or not?

-vova

Comment by Dwight Merriman [ 18/Apr/11 ]

> t.find()

{ "_id" : ObjectId("4dac8e746d2750035ba0e326"), "x" : [ 1, 2, 3 ] } { "_id" : ObjectId("4dac8f376d2750035ba0e327"), "x" : 9 }

{ "_id" : ObjectId("4dac90d16d2750035ba0e328"), "x" :

{ "y" : 3 }

}
> t.find( { x :

{ $type : 3}

} )
{ "_id" : ObjectId("4dac90d16d2750035ba0e328"), "x" :

{ "y" : 3 }

}
> t.find( { x :

{ $type : 4}

} )
> t.find( { x :

{ $type : 1}

} )

{ "_id" : ObjectId("4dac8e746d2750035ba0e326"), "x" : [ 1, 2, 3 ] } { "_id" : ObjectId("4dac8f376d2750035ba0e327"), "x" : 9 }
Comment by Fitz Agard [ 24/Feb/11 ]

Along the lines of the above, is there any way to identify the overal field as an array irrespective of what the values are?

Thanks,
Fitz

Comment by Robert Slifka [ 26/Sep/10 ]

Hi Eliot,

Thanks for the tip! Appreciate the quick response

What is the intended use of type:4?

Rob

Comment by Eliot Horowitz (Inactive) [ 26/Sep/10 ]

This is actually not a bug per se.
Most operators that act on an array look at the elements of the array - and that's what happening here.

Comment by Robert Slifka [ 26/Sep/10 ]

I'm actually seeing them as strings (1.4.4 as well).

> db.type_test.insert(

{rob:['is','cool']}

)
> db.type_test.find()

{ "_id" : ObjectId("4c9fb415691ea2061600bf1d"), "rob" : [ "is", "cool" ] }

> db.type_test.find({'rob':{$type:4}})
> db.type_test.find({'rob':{$type:2}})

{ "_id" : ObjectId("4c9fb415691ea2061600bf1d"), "rob" : [ "is", "cool" ] }

>

Comment by Tatyana Tvardovskaya [ 24/Jul/10 ]

Could be linked to similar bug http://jira.mongodb.org/browse/JAVA-103

Generated at Thu Feb 08 02:57:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.