[SERVER-9639] 2dsphere indexes with {2dsphereIndexVersion:2} should be sparse on geo fields Created: 09/May/13  Updated: 10/Mar/14  Resolved: 20/Feb/14

Status: Closed
Project: Core Server
Component/s: Geo, Index Maintenance
Affects Version/s: 2.4.3
Fix Version/s: 2.6.0-rc0

Type: Bug Priority: Major - P3
Reporter: Johan Hedin Assignee: hari.khalsa@10gen.com
Resolution: Done Votes: 2
Labels: geo, indexing, sparse
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-10709 Invalid BSON docs lead to errors in c... Closed
is duplicated by SERVER-9926 2dsphere index requires field. "Can't... Closed
is duplicated by SERVER-10095 2dsphere index creation fails with nu... Closed
Related
is related to SERVER-11149 Make 2dsphere indexes adhere to norma... Closed
Backwards Compatibility: Major Change
Operating System: ALL
Participants:

 Description   

Starting in 2.6.0-rc0, there are two versions of 2dsphere indexes. All new 2dsphere indexes starting in 2.6.0-rc0 are created with {2dsphereIndexVersion:2}. The V2 behavior is consistent with the behavior of 2d indexes with regards to 'sparseness'

{2dsphereIndexVersion: 1} indexes:

  • Do not support new GeoJSON geometries introduced in 2.5.0: (MultiPoint, MultiLineString, MultiPolygon,
    GeometryCollection).
  • Are not sparse and do not recognize the {sparse: true}

    index modifier.

  • Documents which contain null or "empty" [] for the geo field are rejected and not inserted into the collection.
  • Documents with the geo field undefined are indexed and added to the collection.
  • It is possible to preserve the V1 behavior by indicating the version when creating the index. See SERVER-12175.
    For example, coll.ensureIndex( {geo: "2dsphere"}, {"2dsphereIndexVersion": 1});

{2dsphereIndexVersion: 2} indexes:

  • Ignore the sparse flag and print a warning to the server log if it's set.
  • If a document lacks a 2dsphere-indexed field (or the field is null or an empty array) the document is inserted but not added to the index.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Original Ticket Description
2dsphere indexes does not honor the sparse option. By contrast, 2d indexes are "sparse" by default and require all the fields from the index definition to be present in the document for the document to be included in the index.

Don't know if this is an oversight or by design, but sparse functionality for 2dsphere is usable in the same way as sparse is usable for "normal" indexes.



 Comments   
Comment by Githook User [ 20/Feb/14 ]

Author:

{u'username': u'hkhalsa', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}

Message: SERVER-9639 For 2dsphere V2 indices don't index any docs missing any geo fields

This reverts commit 3b5b35c15690769df2a11a75a3c6e6afd7c2169a.
Branch: master
https://github.com/mongodb/mongo/commit/269c683220692299b5fb9cf431359e3ec299f8fc

Comment by Johan Hedin [ 16/Feb/14 ]

dan@10gen.com, is it possible to resolve this issue with the new 2dsphere index version tag?

See: https://jira.mongodb.org/browse/SERVER-12175

Comment by Pascal Bach [ 05/Jan/14 ]

I got that, I just wanted to make sure if this gets implemented in a later version it also works with empty arrays.

On the other hand. Is there an alternative solution that allows indexing coordinates that are optional?

For the moment I use legacy 2d indexes but I would really like to use the 2dspehre index.

Comment by Daniel Pasette (Inactive) [ 03/Jan/14 ]

pascal-bach, the behavior for 2.6 will be unchanged from the behavior in 2.4.

Comment by Pascal Bach [ 02/Jan/14 ]

Will this also work if the coordinates field is an empty array?
Mongoose for example by default creates empty arrays, they should behave the same way as a null value I suggest.

Comment by Daniel Pasette (Inactive) [ 21/Dec/13 ]

Author:

{u'username': u'hkhalsa', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}

Message: 2dsphere can't be sparse now since it coulnd't be sparse in 2.4
Branch: master
https://github.com/mongodb/mongo/commit/3b5b35c15690769df2a11a75a3c6e6afd7c2169a

Comment by Daniel Pasette (Inactive) [ 21/Dec/13 ]

jhn, We are going to have to revert to the current 2.4 behavior, which means 2dsphere indexes are not sparse by default and ignore the sparse option. In a future version of the server, we will have to bump the version of the index to account for this.

Comment by Johan Hedin [ 20/Dec/13 ]

Dan, do you have any indication on if/how this is going to be resolved? Is it skipped entirely for 2.6?

Comment by Daniel Pasette (Inactive) [ 20/Dec/13 ]

The fix for this needs to be reverted before 2.6. The reason being is that you cannot upgrade a 2.4 server to 2.6 where the 2dsphere index was created with {sparse:true}. Namely, If a document without the geo field set is deleted with a 2.6 server which was indexed in 2.4, it will leave the index record in the index causing corruption.

Comment by Tim van Kasteren [ 17/Sep/13 ]

Note this now works for collections in which entries with a location field are combined with entries where the location field is missing. However, in some of the duplicate posts this bug was reported as not being able to handle null values for the location field. Using null values still results in an error. Ideally an entry is not included in the index when the location field is missing or the location field or its coordinate subfield is null.

Comment by Johan Hedin [ 03/Jul/13 ]

Thanks for the fix Hari!

Comment by hari.khalsa@10gen.com [ 03/Jul/13 ]

I just submitted a change to that makes 2dsphere sparse by default, like 2d. My apologies for the oversight.

Comment by auto [ 03/Jul/13 ]

Author:

{u'username': u'hkhalsa', u'name': u'Hari Khalsa', u'email': u'hkhalsa@10gen.com'}

Message: SERVER-9639 make 2dsphere indices sparse w.r.t. the indexed geo fields
Branch: master
https://github.com/mongodb/mongo/commit/3457954bb082d98dfd769d27771d4e30f3f785a5

Comment by Steven Dunlap [ 28/Jun/13 ]

I would like to "+1" here. We're switching to a 2dsphere index for faster geospacial queries, but a small portion of our data is incomplete (i.e. we don't have the lat/lon data). It would be nice if these were skipped in the index, instead of the whole index refusing the build.

Generated at Thu Feb 08 03:21:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.