Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-28546

Documents can erroneously be unindexed from a partial index

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.2.13, 3.4.4, 3.5.6
    • Affects Version/s: 3.2.12, 3.4.3
    • Component/s: Index Maintenance
    • Labels:
      None
    • Fully Compatible
    • ALL
    • v3.4, v3.2
    • Hide 
      > db.foo.ensureIndex({a:1},{unique:true, partialFilterExpression: {b:1}})
> db.foo.insert({_id:1,a:1})           // This document is not indexed, because it doesn't match the index filter.
WriteResult({ "nInserted" : 1 })
> db.foo.insert({_id:2,a:1,b:1})     // This document is indexed.
WriteResult({ "nInserted" : 1 })
> db.foo.remove({_id:1})            // This erroneously unindexes the _id:2 document.
WriteResult({ "nRemoved" : 1 })
> db.foo.validate(true)               // Index validation detects the problem.
{
	"ns" : "test.foo",
	"nInvalidDocuments" : NumberLong(0),
	"nrecords" : 1,
	"nIndexes" : 2,
	"keysPerIndex" : {
		"test.foo.$_id_" : 1,
		"test.foo.$a_1" : 0
	},
	"indexDetails" : {
		"test.foo.$_id_" : {
			"valid" : true
		},
		"test.foo.$a_1" : {
			"valid" : false
		}
	},
	"valid" : false,
	"warnings" : [ ],
	"errors" : [ ],
	"advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.",
	"ok" : 1
}
>
      Show
      > db.foo.ensureIndex({a:1},{unique:true, partialFilterExpression: {b:1}}) > db.foo.insert({_id:1,a:1}) // This document is not indexed, because it doesn't match the index filter. WriteResult({ "nInserted" : 1 }) > db.foo.insert({_id:2,a:1,b:1}) // This document is indexed. WriteResult({ "nInserted" : 1 }) > db.foo.remove({_id:1}) // This erroneously unindexes the _id:2 document. WriteResult({ "nRemoved" : 1 }) > db.foo.validate(true) // Index validation detects the problem. { "ns" : "test.foo", "nInvalidDocuments" : NumberLong(0), "nrecords" : 1, "nIndexes" : 2, "keysPerIndex" : { "test.foo.$_id_" : 1, "test.foo.$a_1" : 0 }, "indexDetails" : { "test.foo.$_id_" : { "valid" : true }, "test.foo.$a_1" : { "valid" : false } }, "valid" : false, "warnings" : [ ], "errors" : [ ], "advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.", "ok" : 1 } >
    • Storage 2017-04-17

      Issue Status as of Apr 06, 2017

      ISSUE DESCRIPTION
      This issue only affects deployments that:

      • use the WiredTiger storage engine, and
      • remove documents from collections that have at least one index with a {unique: true} specification, and
      • those indexes use a partialFilterExpression

      Deployments that do not meet all three conditions are not affected.

      ISSUE IMPACT
      In affected deployments, when a document that does not pass the partial index filter is removed from a collection, other documents that contain matching keys to the removed document and do pass the partial index filter are erroneously unindexed from the partial index.

      Consequently, queries that utilize this index may not return all stored results. This also has the effect of being able to successfully insert documents that violate the unique key constraint of the index and also pass the index's filter.

      Please see the reproduction steps above for an illustrative example.

      DIAGNOSIS AND AFFECTED VERSIONS
      Indexes created on MongoDB 3.2.0 to 3.2.12 or 3.4.0 to 3.4.3 with the WiredTiger storage engine may be affected.
      Additionally, the index must match both of the following criteria:

      • Is a unique index, e.g. created with {unique: true}
      • Is a partial index, e.g. created with { partialFilterExpression: { foo: bar } }

      To determine if an index matching the criteria above has been affected by this bug, execute db.foo.validate(true). The validation will fail for affected indexes.

      REMEDIATION AND WORKAROUNDS
      The fix is included in the 3.2.13 and 3.4.4 production release. To resolve this issue, affected indexes should be rebuilt after upgrading to a version of MongoDB containing the fix.

      Original description

      This issue affects WiredTiger only. It only affects indexes with {unique: true}, and it only affects indexes using a partialFilterExpression.
      Removing a document that does not pass the filter, and thus is not indexed, will erroneously unindex documents that do pass the filter and contain matching keys to the removed document. See the reproduction steps for an illustrative example.

      This has the effect of being able to successfully insert documents that violate the unique key constraint of the index and also pass the index's filter.

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            milkie@mongodb.com Eric Milkie
            Votes:
            3 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: