[SERVER-28546] Documents can erroneously be unindexed from a partial index Created: 29/Mar/17  Updated: 22/Jan/24  Resolved: 04/Apr/17

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 3.2.12, 3.4.3
Fix Version/s: 3.2.13, 3.4.4, 3.5.6

Type: Bug Priority: Critical - P2
Reporter: Eric Milkie Assignee: Eric Milkie
Resolution: Done Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-28153 Duplicate data is sometimes inserted ... Closed
is duplicated by SERVER-28516 index corruption Closed
is duplicated by SERVER-25791 Unique index becomes inconsistent wit... Closed
Problem/Incident
causes SERVER-32001 unindexing a key in a partial unique ... Closed
Related
related to SERVER-75922 Partial unique indexes created on Mon... Closed
related to SERVER-85536 [4.4] removing unindexed unique parti... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.4, v3.2
Steps To Reproduce:

> db.foo.ensureIndex({a:1},{unique:true, partialFilterExpression: {b:1}})
> db.foo.insert({_id:1,a:1})           // This document is not indexed, because it doesn't match the index filter.
WriteResult({ "nInserted" : 1 })
> db.foo.insert({_id:2,a:1,b:1})     // This document is indexed.
WriteResult({ "nInserted" : 1 })
> db.foo.remove({_id:1})            // This erroneously unindexes the _id:2 document.
WriteResult({ "nRemoved" : 1 })
> db.foo.validate(true)               // Index validation detects the problem.
{
	"ns" : "test.foo",
	"nInvalidDocuments" : NumberLong(0),
	"nrecords" : 1,
	"nIndexes" : 2,
	"keysPerIndex" : {
		"test.foo.$_id_" : 1,
		"test.foo.$a_1" : 0
	},
	"indexDetails" : {
		"test.foo.$_id_" : {
			"valid" : true
		},
		"test.foo.$a_1" : {
			"valid" : false
		}
	},
	"valid" : false,
	"warnings" : [ ],
	"errors" : [ ],
	"advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.",
	"ok" : 1
}
> 

Sprint: Storage 2017-04-17
Participants:
Case:

 Description   
Issue Status as of Apr 06, 2017

ISSUE DESCRIPTION
This issue only affects deployments that:

  • use the WiredTiger storage engine, and
  • remove documents from collections that have at least one index with a {unique: true} specification, and
  • those indexes use a partialFilterExpression

Deployments that do not meet all three conditions are not affected.

ISSUE IMPACT
In affected deployments, when a document that does not pass the partial index filter is removed from a collection, other documents that contain matching keys to the removed document and do pass the partial index filter are erroneously unindexed from the partial index.

Consequently, queries that utilize this index may not return all stored results. This also has the effect of being able to successfully insert documents that violate the unique key constraint of the index and also pass the index's filter.

Please see the reproduction steps above for an illustrative example.

DIAGNOSIS AND AFFECTED VERSIONS
Indexes created on MongoDB 3.2.0 to 3.2.12 or 3.4.0 to 3.4.3 with the WiredTiger storage engine may be affected.
Additionally, the index must match both of the following criteria:

  • Is a unique index, e.g. created with {unique: true}
  • Is a partial index, e.g. created with { partialFilterExpression: { foo: bar } }

To determine if an index matching the criteria above has been affected by this bug, execute db.foo.validate(true). The validation will fail for affected indexes.

REMEDIATION AND WORKAROUNDS
The fix is included in the 3.2.13 and 3.4.4 production release. To resolve this issue, affected indexes should be rebuilt after upgrading to a version of MongoDB containing the fix.

Original description

This issue affects WiredTiger only. It only affects indexes with {unique: true}, and it only affects indexes using a partialFilterExpression.
Removing a document that does not pass the filter, and thus is not indexed, will erroneously unindex documents that do pass the filter and contain matching keys to the removed document. See the reproduction steps for an illustrative example.

This has the effect of being able to successfully insert documents that violate the unique key constraint of the index and also pass the index's filter.



 Comments   
Comment by Githook User [ 05/Apr/17 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-28546 confirm WiredTiger unindex operations remove the correct document

Previously, unindex operations on WiredTiger would not confirm the record id matches.
This could result in erroroneous entries being removed from the index, for the case where
an index with a unique constraint also had a partial filter expression.

(cherry picked from commit 052345f6c592b1b5e626e05ed8c59e9b20b6d7ee)
Branch: v3.2
https://github.com/mongodb/mongo/commit/1914a8164e8dd5d9a8d326b2c281f5fff87cd4e3

Comment by Githook User [ 05/Apr/17 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-28546 confirm WiredTiger unindex operations remove the correct document

Previously, unindex operations on WiredTiger would not confirm the record id matches.
This could result in erroroneous entries being removed from the index, for the case where
an index with a unique constraint also had a partial filter expression.

(cherry picked from commit 052345f6c592b1b5e626e05ed8c59e9b20b6d7ee)
Branch: v3.4
https://github.com/mongodb/mongo/commit/f8d91dfec1413570a6b9ec2d4571592f9202529e

Comment by Githook User [ 04/Apr/17 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-28546 confirm WiredTiger unindex operations remove the correct document

Previously, unindex operations on WiredTiger would not confirm the record id matches.
This could result in erroroneous entries being removed from the index, for the case where
an index with a unique constraint also had a partial filter expression.
Branch: master
https://github.com/mongodb/mongo/commit/052345f6c592b1b5e626e05ed8c59e9b20b6d7ee

Generated at Thu Feb 08 04:18:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.