[SERVER-75922] Partial unique indexes created on MongoDB 4.0 can be missing index keys after upgrade to 4.2 and later, leading to uniqueness violations Created: 10/Apr/23 Updated: 22/Jan/24 Resolved: 27/Apr/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | 4.2.0, 4.4.0, 5.0.0, 6.0.0, 6.3.0-rc3 |
| Fix Version/s: | 7.1.0-rc0, 6.0.7, 5.0.19, 4.4.23, 7.0.0-rc3 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Max Hirschhorn | Assignee: | Dianna Hohensee (Inactive) |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v7.0, v6.3, v6.0, v5.0, v4.4, v4.2
|
||||||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2023-04-17, Execution Team 2023-05-01 | ||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||
| Description |
|
Issue Status as of Dec 04, 2023 ISSUE DESCRIPTION AND IMPACT Users who upgraded from MongoDB 4.0 to 4.2+ (featureCompatibility 4.2+) may experience index inconsistencies within unique partial indexes (unique indexes which specify a partialFilterExpression), in the form of missing index keys. Unique primary (_id) indexes are not affected. Affected versions of MongoDB incorrectly remove index entries from the unique partial index when all of the following take place:
When the un-indexed document is deleted, affected versions of MongoDB incorrectly delete the key for the indexed document. Missing index entries in unique partial index in v4.2+ can have the following effects:
DIAGNOSIS AND AFFECTED VERSIONS Affected Versions: 4.2.0, 4.4.0, 5.0.0, 6.0.0, 6.3.0-rc3 Users who upgraded to v4.2 and are still running in FCV 4.0 are not impacted. Even after upgrading to a fixed version, users can still be impacted from missing index entries and documents with duplicate index keys. In FCV 4.2+, users who have collections that rely on unique indexes with partialFilterExpression may be impacted by this bug. Users can verify if a unique partial index is set by calling `getIndexes` on all the collections. Missing unique index entries can be checked by calling validate. REMEDIATION AND WORKAROUNDS User action is required in order to remediate this issue. Impacted users should follow the below steps:
Rebuilding the index may fail due to existing documents with duplicate keys in the collection.
Original descriptionThe index format for unique indexes changed between MongoDB 4.0 and 4.2.
In FCV 4.2 and greater, a mongod writes the new format for index entries while still supporting the ability to read both formats. The in-place conversion incorrectly assumes that when the document is being unindexed the index entry in the 4.0 format can always be removed. This assumption results in documents having missing index keys in a very similar way to |
| Comments |
| Comment by Noopur Gupta [ 04/Dec/23 ] | |||||||||||||||||||||||||||||
|
Issue Status as of Dec 04, 2023 ISSUE DESCRIPTION AND IMPACT Users who upgraded from MongoDB 4.0 to 4.2+ (featureCompatibility 4.2+) may experience index inconsistencies within unique partial indexes (unique indexes which specify a partialFilterExpression), in the form of missing index keys. Unique primary (_id) indexes are not affected. Affected versions of MongoDB incorrectly remove index entries from the unique partial index when all of the following take place:
When the un-indexed document is deleted, affected versions of MongoDB incorrectly delete the key for the indexed document. Missing index entries in unique partial index in v4.2+ can have the following effects:
DIAGNOSIS AND AFFECTED VERSIONS Affected Versions: 4.2.0, 4.4.0, 5.0.0, 6.0.0, 6.3.0-rc3 Users who upgraded to v4.2 and are still running in FCV 4.0 are not impacted. Even after upgrading to a fixed version, users can still be impacted from missing index entries and documents with duplicate index keys. In FCV 4.2+, users who have collections that rely on unique indexes with partialFilterExpression may be impacted by this bug. Users can verify if a unique partial index is set by calling `getIndexes` on all the collections. Missing unique index entries can be checked by calling validate. REMEDIATION AND WORKAROUNDS User action is required in order to remediate this issue. Impacted users should follow the below steps:
Rebuilding the index may fail due to existing documents with duplicate keys in the collection.
| |||||||||||||||||||||||||||||
| Comment by Githook User [ 26/May/23 ] | |||||||||||||||||||||||||||||
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}Message: | |||||||||||||||||||||||||||||
| Comment by Githook User [ 24/May/23 ] | |||||||||||||||||||||||||||||
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}Message: (cherry picked from commit 13bcde88db4ec79bbf63053c72bc60bb4ab424b5) | |||||||||||||||||||||||||||||
| Comment by Githook User [ 24/May/23 ] | |||||||||||||||||||||||||||||
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}Message: (cherry picked from commit 13bcde88db4ec79bbf63053c72bc60bb4ab424b5) | |||||||||||||||||||||||||||||
| Comment by Githook User [ 24/May/23 ] | |||||||||||||||||||||||||||||
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}Message: (cherry picked from commit 13bcde88db4ec79bbf63053c72bc60bb4ab424b5) | |||||||||||||||||||||||||||||
| Comment by Githook User [ 26/Apr/23 ] | |||||||||||||||||||||||||||||
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@mongodb.com', 'username': 'DiannaHohensee'}Message: |