Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22970

Background index build produces an index with mismatched index keys and documents

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      This bug appeared a few times when running jstests/noPassthroughWithMongod/index_multi.js on a standalone mongod with WiredTiger.

      Show
      This bug appeared a few times when running jstests/noPassthroughWithMongod/index_multi.js on a standalone mongod with WiredTiger.
    • Sprint:
      Integration 12 (04/04/16), Integration 13 (04/22/16)

      Description

      Issue Status as of May 4, 2016

      ISSUE SUMMARY
      While a background index build is in progress, document updates modifying fields contained in the the index specification may, under specific circumstances, cause mismatched index entries to appear.

      This bug may affect background-built index builds only, and only in combination with update operations.

      USER IMPACT
      This bug may affect the behavior of queries that use an affected index, which may return incorrect results. Symptoms that indicate a background-built index is affected by this bug include:

      • find() queries covered by the index may:
        • return documents that don't match the query predicate
        • display documents that no longer exist in the collection
        • display more than one document with the same _id field
      • count() commands may return incorrect counts
      • remove() commands may remove documents that don't match the query predicate

      Operations that do not use affected indexes are not impacted by this bug.

      Note that collection data consistency is unaffected by this bug, only index entries may be affected.

      WORKAROUNDS
      There are no workarounds for this issue. While not all deployments using background-built indexes will be impacted, all users running the affected configurations should upgrade to a version containing the fix and rebuild all background-built indexes to make sure they're not impacted by this bug.

      AFFECTED VERSIONS

      • MongoDB 3.0 with the WiredTiger storage engine
        While a background index build is in progress, any document updates modifying fields contained in the the index specification that happen at the same time as the background index build is reading those same documents will result in mismatched index entries. Technical details about this but can be found in SERVER-22970 and SERVER-23807.
      • MongoDB 3.2 with MMAPv1 and WiredTiger storage engines
        While a background index build on a set of fields is in progress, any document updates modifying fields contained in the index specification may result in mismatched index entries if the background index build has already processed the documents to be updated, or if such documents are being processed by the index build at the same time. Technical details about this bug can be found in SERVER-22970, SERVER-23807 and SERVER-23655.

      FIX VERSION
      The fix is included in the 3.2.6 production release. A fix for MongoDB 3.0 for users of the WiredTiger engine is included in the 3.0.12 production release.

      After upgrading to one of these versions, all background-built indexes need to be rebuilt to avoid this issue.

      Original description

      After doing a background index build, it's possible for there to be more index keys than documents in a compound, non-multikey index.

      > db.index_multi.find().count()
      10000
      > db.index_multi.find().hint("field52_1_field53_1").count()
      10001
      

      This is caused by multiple index entries pointing to the same document.

      > db.index_multi.find({_id: 5468}, {_id:1}).showRecordId()
      { "_id" : 5468, "$recordId" : NumberLong(5469) }
      > db.index_multi.find({_id: 5468}, {_id:1}).hint("field52_1_field53_1").showRecordId()
      { "_id" : 5468, "$recordId" : NumberLong(5469) }
      { "_id" : 5468, "$recordId" : NumberLong(5469) }
      

      Index spec:

          {
              "v" : 1,
              "key" : {
                  "field52" : 1,
                  "field53" : 1
              },
              "name" : "field52_1_field53_1",
              "ns" : "test.index_multi",
              "background" : true
          }
      

      Here is a summary of affected versions and storage engines (where "yes" means we have been able to reproduce the index corruption):

      version MMAPv1 WiredTiger
      2.6.12 No -
      3.0.11 No Yes
      3.2.5 Yes Yes
      3.3.4 Yes Yes

        Issue Links

          Activity

          Hide
          robert.guo Robert Guo added a comment -

          jstests/noPassthroughWithMongod/index_multi.js will be disabled after SERVER-7326 is pushed. Please re-enable the test in no_passthrough_with_mongod.yml after this ticket is fixed.

          Show
          robert.guo Robert Guo added a comment - jstests/noPassthroughWithMongod/index_multi.js will be disabled after SERVER-7326 is pushed. Please re-enable the test in no_passthrough_with_mongod.yml after this ticket is fixed.
          Hide
          milkie Eric Milkie added a comment -

          I would file a separate ticket just for that task, so we don't forget.

          Show
          milkie Eric Milkie added a comment - I would file a separate ticket just for that task, so we don't forget.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

          Message: SERVER-22970 fix update race with background index build

          There are two distinct fixes that must be done together:

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'} Message: SERVER-22970 fix update race with background index build There are two distinct fixes that must be done together: SERVER-23807 : update should throw write conflict on unindex SERVER-18844 : background index builds should reacquire the snapshot to properly detect write conflicts with concurrent updates Branch: master https://github.com/mongodb/mongo/commit/f27b0cac4869fa506c6ed6f0dfc885b9edcd765a
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

          Message: SERVER-22970 fix update race with background index build

          There are two distinct fixes that must be done together:

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'} Message: SERVER-22970 fix update race with background index build There are two distinct fixes that must be done together: SERVER-23807 : update should throw write conflict on unindex SERVER-18844 : background index builds should reacquire the snapshot to properly detect write conflicts with concurrent updates Branch: v3.2 https://github.com/mongodb/mongo/commit/be81cc9e83d09d4dc206c59656fc7e51c3e4fc12
          Hide
          kyle.suarez Kyle Suarez added a comment -

          Author:

          {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

          Message: SERVER-22970 index build must X-lock collection
          Index builds must lock the collection with an X lock, rather than an IX lock, to
          prevent a race with concurrent updates.
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/dc729bf1e79318b17bcb02550ce7a0a51b814d72

          Show
          kyle.suarez Kyle Suarez added a comment - Author: {u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'} Message: SERVER-22970 index build must X-lock collection Index builds must lock the collection with an X lock, rather than an IX lock, to prevent a race with concurrent updates. Branch: v3.0 https://github.com/mongodb/mongo/commit/dc729bf1e79318b17bcb02550ce7a0a51b814d72
          Hide
          vtomasr5 Vicenç Juan Tomàs added a comment -

          Hi,

          We are running a big sharded cluster with 5 shards (3 nodes each rs) and we have a large number of background indexes created. We've updated to 3.0.12 and we've been thinking to rebuild the indexes due to this bug one by one. The size of the majority of the collections makes impossible to rebuild the indexes without affecting the business.

          We know that a full data re-sync its doable and it regenerates all the indexes. My question is: Will a full resync fix the indexes created in background mode?

          Thanks!

          Show
          vtomasr5 Vicenç Juan Tomàs added a comment - Hi, We are running a big sharded cluster with 5 shards (3 nodes each rs) and we have a large number of background indexes created. We've updated to 3.0.12 and we've been thinking to rebuild the indexes due to this bug one by one. The size of the majority of the collections makes impossible to rebuild the indexes without affecting the business. We know that a full data re-sync its doable and it regenerates all the indexes. My question is: Will a full resync fix the indexes created in background mode? Thanks!
          Hide
          ramon.fernandez Ramon Fernandez added a comment -

          Vicenç Juan Tomàs, yes, a full initial sync of a node in a replica set will rebuild all indexes on that node and address the issue described in this ticket. Note that you'll need to resync all 15 nodes (5 shards x 3 rs/shard) to rebuild the background indexes in all nodes. You may also want to consider this other procedure to rebuild your indexes if your oplog is big enough.

          For additional support-related discussion please post on the mongodb-user group. See also our Technical Support page for additional support resources.

          Thanks,
          Ramón.

          Show
          ramon.fernandez Ramon Fernandez added a comment - Vicenç Juan Tomàs , yes, a full initial sync of a node in a replica set will rebuild all indexes on that node and address the issue described in this ticket. Note that you'll need to resync all 15 nodes (5 shards x 3 rs/shard) to rebuild the background indexes in all nodes. You may also want to consider this other procedure to rebuild your indexes if your oplog is big enough. For additional support-related discussion please post on the mongodb-user group . See also our Technical Support page for additional support resources. Thanks, Ramón.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile