[SERVER-28975] Cannot remove document with 2dsphere partialFilter index Created: 26/Apr/17  Updated: 30/Oct/23  Resolved: 08/Feb/18

Status: Closed
Project: Core Server
Component/s: Geo, Index Maintenance
Affects Version/s: 3.4.4
Fix Version/s: 3.7.2

Type: Bug Priority: Major - P3
Reporter: Andrei Belashou Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-39705 IndexBuildInterceptor does not faithf... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

// Create collection with partial 2dsphere index (condition is geometry_status >= 0) 
db.test.createIndex(
    {"geoJson": "2dsphere"},
    { partialFilterExpression: {"geometry_status": {$gte:0}}}
);
// Insert document that doesn't meet partial index condition
db.test.insert({
		"_id": ObjectId('58f8e5e06758c605c09e9110'),
		"geometry_status" : -1,
        "geoJson" : {
            "type" : "Polygon",
            "coordinates" : [ 
                [ 
                    [ 
                        5.86121301988361, 
                        51.9128247961245
                    ], 
                    [ 
                        5.86121303441693, 
                        51.9128247960658
                    ], 
                    [ 
                        5.86121306203623, 
                        51.9128260363524
                    ], 
                    [ 
                        5.86077631455415, 
                        51.9128299191492
                    ], 
                    [ 
                        5.86076411163462, 
                        51.9122347660331
                    ], 
                    [ 
                        5.8612026892498, 
                        51.9122313253
                    ], 
                    [ 
                        5.8612136159931, 
                        51.912764202971
                    ], 
                    [ 
                        5.8613461964441, 
                        51.9127630207468
                    ], 
                    [ 
                        5.86134760592148, 
                        51.9128235878296
                    ], 
                    [ 
                        5.86121486552018, 
                        51.9128247796886
                    ], 
                    [ 
                        5.86121303441693, 
                        51.9128247960658
                    ], 
                    [ 
                        5.86121301988361, 
                        51.9128247961245
                    ]
                ]
            ]
        }});
// Remove this document
db.test.remove({"_id": ObjectId('58f8e5e06758c605c09e9110')});
// Error

Sprint: Storage 2018-02-12
Participants:
Case:

 Description   

Hello,

unexpected error occurred while trying to remove document that contains invalid geoJson from collection with partial index:

WriteResult({
	"nRemoved" : 0,
	"writeError" : {
		"code" : 16755,
		"errmsg" : "Can't extract geo keys: { _id: ObjectId('58f8e5e06758c605c09e9110'), geometry_status: -1.0, geoJson: { type: \"Polygon\", coordinates: [ [ [ 5.86121301988361, 51.9128247961245 ], [ 5.86121303441693, 51.9128247960658 ], [ 5.86121306203623, 51.9128260363524 ], [ 5.86077631455415, 51.9128299191492 ], [ 5.86076411163462, 51.9122347660331 ], [ 5.8612026892498, 51.9122313253 ], [ 5.8612136159931, 51.912764202971 ], [ 5.8613461964441, 51.9127630207468 ], [ 5.86134760592148, 51.9128235878296 ], [ 5.86121486552018, 51.9128247796886 ], [ 5.86121303441693, 51.9128247960658 ], [ 5.86121301988361, 51.9128247961245 ] ] ] } }  Loop is not valid: [ [ 5.86121301988361, 51.9128247961245 ], [ 5.86121303441693, 51.9128247960658 ], [ 5.86121306203623, 51.9128260363524 ], [ 5.86077631455415, 51.9128299191492 ], [ 5.86076411163462, 51.9122347660331 ], [ 5.8612026892498, 51.9122313253 ], [ 5.8612136159931, 51.912764202971 ], [ 5.8613461964441, 51.9127630207468 ], [ 5.86134760592148, 51.9128235878296 ], [ 5.86121486552018, 51.9128247796886 ], [ 5.86121303441693, 51.9128247960658 ], [ 5.86121301988361, 51.9128247961245 ] ] Duplicate vertices: 1 and 10"
	}
})

Partial index is 2dsphere with condition geometry_status >= 0. Inserted document has invalid goeJson but geometry_status = -1 so it can be inserted. But cannot be removed.



 Comments   
Comment by Githook User [ 08/Feb/18 ]

Author:

{'email': 'louis.williams@mongodb.com', 'name': 'Louis Williams', 'username': 'louiswilliams'}

Message: SERVER-28975 Fix typos
Branch: master
https://github.com/mongodb/mongo/commit/1e121053951d4b5dca898f4b7939d0a98d7bde09

Comment by Louis Williams [ 08/Feb/18 ]

The following commit was made to master with the wrong ticket in the commit message:

Message: SERVER-29876 Check if partial filter applies when unindexing documents
Branch: master
https://github.com/mongodb/mongo/commit/b77e46685dc7ae8b9893ef019a1cbc67798bb749

Comment by Ian Whalen (Inactive) [ 16/Oct/17 ]

louis.williams we're putting this back on the backlog just to make sure it doesn't get lost from the Storage team's view, but please ping me or Eric if you do get spare time and come up with a patch for this.

Comment by Louis Williams [ 13/Oct/17 ]

After talking with max.hirschhorn and david.storch use the kRelaxConstraints constant to ignore whitelisted errors when calling IndexAccessMethod::getKeys(). If a whitelisted error is thrown, but the partialIndexFilter applies to the document, then the error will be thrown.

Comment by Louis Williams [ 12/Oct/17 ]

It seems like these are the only potential solutions. max.hirschhorn and david.storch let me know if you agree. Solution 3 is a much larger project.

1. On unindex(), check if the partialIndex filter applies to the document. If it doesn't, don't attempt to unindex it.

  • Problem: If the semantics of filter matching change in a future release, an index could be left never deleted on removal, dangling.

2. On removal, use the existing kRelaxConstraints constant to ignore all key errors when calling IndexAccessMethod::getKeys().

  • Problem: An error retrieving one key for a document in the set will prevent the retrieval, and subsequent unindexing of keys for other later valid documents.

3. On remove, use the existing kRelaxConstraints, to ignore individual key errors, and pass to the IndexAccessMethod::doGetKeys() virtual function, which all subclasses must implement.

  • This means all key retrieval errors must be handled on a document-by-document basis, and ignored explicitly when kRelaxConstraints is set. If an error is encountered, the retrieval of the key is aborted, but the next documents are still checked to ensure that no keys are skipped.
Comment by Eric Milkie [ 26/Apr/17 ]

That's a better workaround. I wasn't sure that that method would work (it may have given a similar error). Thanks for the tip!

Comment by Andrei Belashou [ 26/Apr/17 ]

Hi Eric,
thanks for reply. We also found out another way to remove document. It is possible to update document and set any valid indexable geoJson and then remove the document. In this case we don't need to drop and create index. Might be helpful for someone.

Comment by Eric Milkie [ 26/Apr/17 ]

Hi Andrei,
Thanks for filing this bug report. We will try to fix it in a future minor release of 3.4, but in the meantime, the only way I can think of to be able to remove the document is to drop the geo index, delete the document, and then add the index back.

Generated at Thu Feb 08 04:19:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.