[SERVER-41213] Unique background index builds may produce inconsistent keys Created: 17/May/19  Updated: 29/Oct/23  Resolved: 21/May/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 4.0.6
Fix Version/s: 3.4.21, 3.6.13, 4.0.10, 4.1.12

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-41803 Standalone server crash with 'Unique ... Closed
Related
related to SERVER-22970 Background index build produces an in... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6, v3.4, v3.2
Steps To Reproduce:

mongo --port 29523 --eval 'var array = []; for (i = 0; i < 1000; i++) { array.push(i); } for (j = 0; j < 25; j++) { db.hugeindex.insert({_id: j, a: array}); } db.hugeindex.createIndex({a: 1}, {background: true});'
mongo --port 29523 --eval 'db.hugeindex.createIndex({a: 1, _id: 1}, {unique: true, background: true});' &
mongo --port 29523 --eval 'db.hugeindex.createIndex({_id: 1, a: 1}, {unique: true, background: true});' &
 
mongo --port 29523 --eval 'for (j = 0; j < 25; j++) { try { db.hugeindex.findOneAndUpdate({_id: j}, {$pop: {"a": -1}}); db.hugeindex.findOneAndUpdate({_id: j}, {$pop: {"a": 1}}); }catch(exc) {} }' &
 
mongo --port 29523 --eval 'for (j = 0; j < 3; j++) { try { db.hugeindex.updateOne({_id: j * 10}, {$set: {a: 1}}); db.hugeindex.deleteOne({_id: j * 19}); }catch(exc) {} }' &

Sprint: Storage NYC 2019-05-20, Execution Team 2019-06-03
Participants:

 Description   
Issue Status as of May 23, 2019

ISSUE DESCRIPTION AND IMPACT
Background, unique, non-partial indexes on WiredTiger may build indexes that have mismatched keys. As a result, there may be index keys that point to documents without respective matching keys. It is possible that query results that use such an index may return invalid results. Despite this issue, the uniqueness constraint of the index is still maintained and no documents in the collection will violate the constraint.

DIAGNOSIS AND AFFECTED VERSIONS
Any user running WiredTiger who has created a background, unique, non-partial index on a non-empty collection prior to MongoDB 3.4.21, 3.6.13, or 4.0.10 while performing deletes or updates to the indexed fields during the index build may have been affected. Users can check the settings used to create an index by calling getIndexes on all of their collections.

Users who meet the above criteria should run validate against their collection to determine whether their indexes are invalid and need to be rebuilt on a version containing the fix for this issue. In MongoDB 3.4 and before, users should validate with the {full:true} option passed. In MongoDB 3.6 and 4.0, users need not specify {full:true} when invoking the validate command to identify this issue.

REMEDIATION AND WORKAROUNDS
The fix is included in the 3.4.21, 3.6.13, 4.0.10 production release. Affected users should upgrade to these versions and rebuild the invalid indexes.

Original description

{
	"ns" : "test.hugeindex",
	"nInvalidDocuments" : NumberLong(0),
	"nrecords" : 23,
	"nIndexes" : 4,
	"keysPerIndex" : {
		"_id_" : 23,
		"a_1" : 20960,
		"a_1__id_1" : 20960,
		"_id_1_a_1" : 22958
	},
	"valid" : false,
	"warnings" : [
		"Detected 1998 extra index entries.",
		"Some checks omitted for speed. use {full:true} option to do more thorough scan."
	],
	"errors" : [
		"Index with name '_id_1_a_1' has inconsistencies."
	],
	"extraIndexEntries" : [
		{
			"indexName" : "_id_1_a_1",
			"recordId" : NumberLong(11),
			"indexKey" : {
				"_id" : 10,
				"a" : 0
			}
		},
		{
			"indexName" : "_id_1_a_1",
			"recordId" : NumberLong(11),
			"indexKey" : {
				"_id" : 10,
				"a" : 2
			}
		},
		{
			"indexName" : "_id_1_a_1",
			"recordId" : NumberLong(11),
			"indexKey" : {
				"_id" : 10,
				"a" : 3
			}
		},
<snip>
		{
			"indexName" : "_id_1_a_1",
			"recordId" : NumberLong(21),
			"indexKey" : {
				"_id" : 20,
				"a" : 998
			}
		},
		{
			"indexName" : "_id_1_a_1",
			"recordId" : NumberLong(21),
			"indexKey" : {
				"_id" : 20,
				"a" : 999
			}
		}
	],
	"missingIndexEntries" : [ ],
	"advice" : "A corrupt namespace has been detected. See http://dochub.mongodb.org/core/data-recovery for recovery steps.",
	"ok" : 1
}



 Comments   
Comment by Githook User [ 21/May/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-41213: Trigger a write conflict in unique, non-partial index build code path.

(cherry picked from commit 3372bc595d099392e422d735074ba687efd6634b)
Branch: v3.4
https://github.com/mongodb/mongo/commit/ba2e818de44c9a331ce09a624780a2bd841cf6e4

Comment by Githook User [ 21/May/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-41213: Trigger a write conflict in unique, non-partial index build code path.

(cherry picked from commit 3372bc595d099392e422d735074ba687efd6634b)
Branch: v3.6
https://github.com/mongodb/mongo/commit/1d31cf9ac0b6a3eef3d0b290ec0333f589968cd0

Comment by Githook User [ 21/May/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-41213: Trigger a write conflict in unique, non-partial index build code path.

(cherry picked from commit 3372bc595d099392e422d735074ba687efd6634b)
Branch: v4.0
https://github.com/mongodb/mongo/commit/7afd28d78040c3d171cc9711d419e1294404b267

Comment by Githook User [ 21/May/19 ]

Author:

{'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb', 'username': 'dgottlieb'}

Message: SERVER-41213: Trigger a write conflict in unique, non-partial index build code path.
Branch: master
https://github.com/mongodb/mongo/commit/3372bc595d099392e422d735074ba687efd6634b

Generated at Thu Feb 08 04:57:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.