[SERVER-22970] Background index build produces an index with mismatched index keys and documents Created: 04/Mar/16  Updated: 20/May/19  Resolved: 21/Apr/16

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: 3.0.11, 3.2.5, 3.3.4
Fix Version/s: 3.0.12, 3.2.6, 3.3.5

Type: Bug Priority: Blocker - P1
Reporter: Robert Guo (Inactive) Assignee: Kyle Suarez
Resolution: Done Votes: 0
Labels: code-only, fuzzer-blocker
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-23655 CollectionInfoCache is not invalidate... Closed
depends on SERVER-23807 Updates should always throw WriteConf... Closed
is depended on by SERVER-23695 re-enable index_multi.js and indexbg_... Closed
Duplicate
Related
is related to SERVER-41213 Unique background index builds may pr... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

This bug appeared a few times when running jstests/noPassthroughWithMongod/index_multi.js on a standalone mongod with WiredTiger.

Sprint: Integration 12 (04/04/16), Integration 13 (04/22/16)
Participants:
Linked BF Score: 0

 Description   
Issue Status as of May 4, 2016

ISSUE SUMMARY
While a background index build is in progress, document updates modifying fields contained in the the index specification may, under specific circumstances, cause mismatched index entries to appear.

This bug may affect background-built index builds only, and only in combination with update operations.

USER IMPACT
This bug may affect the behavior of queries that use an affected index, which may return incorrect results. Symptoms that indicate a background-built index is affected by this bug include:

  • find() queries covered by the index may:
    • return documents that don't match the query predicate
    • display documents that no longer exist in the collection
    • display more than one document with the same _id field
  • count() commands may return incorrect counts
  • remove() commands may remove documents that don't match the query predicate

Operations that do not use affected indexes are not impacted by this bug.

Note that collection data consistency is unaffected by this bug, only index entries may be affected.

WORKAROUNDS
There are no workarounds for this issue. While not all deployments using background-built indexes will be impacted, all users running the affected configurations should upgrade to a version containing the fix and rebuild all background-built indexes to make sure they're not impacted by this bug.

AFFECTED VERSIONS

  • MongoDB 3.0 with the WiredTiger storage engine
    While a background index build is in progress, any document updates modifying fields contained in the the index specification that happen at the same time as the background index build is reading those same documents will result in mismatched index entries. Technical details about this but can be found in SERVER-22970 and SERVER-23807.
  • MongoDB 3.2 with MMAPv1 and WiredTiger storage engines
    While a background index build on a set of fields is in progress, any document updates modifying fields contained in the index specification may result in mismatched index entries if the background index build has already processed the documents to be updated, or if such documents are being processed by the index build at the same time. Technical details about this bug can be found in SERVER-22970, SERVER-23807 and SERVER-23655.

FIX VERSION
The fix is included in the 3.2.6 production release. A fix for MongoDB 3.0 for users of the WiredTiger engine is included in the 3.0.12 production release.

After upgrading to one of these versions, all background-built indexes need to be rebuilt to avoid this issue.

Original description

After doing a background index build, it's possible for there to be more index keys than documents in a compound, non-multikey index.

> db.index_multi.find().count()
10000
> db.index_multi.find().hint("field52_1_field53_1").count()
10001

This is caused by multiple index entries pointing to the same document.

> db.index_multi.find({_id: 5468}, {_id:1}).showRecordId()
{ "_id" : 5468, "$recordId" : NumberLong(5469) }
> db.index_multi.find({_id: 5468}, {_id:1}).hint("field52_1_field53_1").showRecordId()
{ "_id" : 5468, "$recordId" : NumberLong(5469) }
{ "_id" : 5468, "$recordId" : NumberLong(5469) }

Index spec:

    {
        "v" : 1,
        "key" : {
            "field52" : 1,
            "field53" : 1
        },
        "name" : "field52_1_field53_1",
        "ns" : "test.index_multi",
        "background" : true
    }

Here is a summary of affected versions and storage engines (where "yes" means we have been able to reproduce the index corruption):

version MMAPv1 WiredTiger
2.6.12 No -
3.0.11 No Yes
3.2.5 Yes Yes
3.3.4 Yes Yes


 Comments   
Comment by Ramon Fernandez Marina [ 07/Jun/16 ]

vtomasr5, yes, a full initial sync of a node in a replica set will rebuild all indexes on that node and address the issue described in this ticket. Note that you'll need to resync all 15 nodes (5 shards x 3 rs/shard) to rebuild the background indexes in all nodes. You may also want to consider this other procedure to rebuild your indexes if your oplog is big enough.

For additional support-related discussion please post on the mongodb-user group. See also our Technical Support page for additional support resources.

Thanks,
Ramón.

Comment by Vicenç Juan Tomàs [ 07/Jun/16 ]

Hi,

We are running a big sharded cluster with 5 shards (3 nodes each rs) and we have a large number of background indexes created. We've updated to 3.0.12 and we've been thinking to rebuild the indexes due to this bug one by one. The size of the majority of the collections makes impossible to rebuild the indexes without affecting the business.

We know that a full data re-sync its doable and it regenerates all the indexes. My question is: Will a full resync fix the indexes created in background mode?

Thanks!

Comment by Kyle Suarez [ 29/Apr/16 ]

Author:

{u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

Message: SERVER-22970 index build must X-lock collection
Index builds must lock the collection with an X lock, rather than an IX lock, to
prevent a race with concurrent updates.
Branch: v3.0
https://github.com/mongodb/mongo/commit/dc729bf1e79318b17bcb02550ce7a0a51b814d72

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

Message: SERVER-22970 fix update race with background index build

There are two distinct fixes that must be done together:

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'ksuarz', u'name': u'Kyle Suarez', u'email': u'ksuarz@gmail.com'}

Message: SERVER-22970 fix update race with background index build

There are two distinct fixes that must be done together:

Comment by Eric Milkie [ 13/Apr/16 ]

I would file a separate ticket just for that task, so we don't forget.

Comment by Robert Guo (Inactive) [ 13/Apr/16 ]

jstests/noPassthroughWithMongod/index_multi.js will be disabled after SERVER-7326 is pushed. Please re-enable the test in no_passthrough_with_mongod.yml after this ticket is fixed.

Generated at Thu Feb 08 04:01:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.