[SERVER-61427] Unique index builds can cause a loss of availability during commit due to checking many false duplicates Created: 11/Nov/21 Updated: 29/Oct/23 Resolved: 18/Nov/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2.0, 4.4.0, 5.0.0 |
| Fix Version/s: | 5.2.0, 5.1.2, 5.0.6, 4.4.11, 4.2.18 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Gregory Noma | Assignee: | Gregory Noma |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Backport Requested: |
v5.1, v5.0, v4.4, v4.2
|
||||||||||||
| Sprint: | Execution Team 2021-11-29 | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
While an index build is in progress, we take any concurrent writes and insert them into the side writes table. Once the collection scan and bulk load, we drain the writes from the side writes table into the index. However, it is possible for these concurrent writes to also be seen by the collection scan. In this case, we end up inserting the same index key twice: once during collection scan/bulk load, and once while draining side writes. This is not a correctness issue; for non-unique indexes we simply ignore that the key already exists, and for unique indexes we write down the key as a potential duplicate. For the unique case, we then check all of these false "duplicates" under a MODE_X collection lock before committing the index. This can lead to a loss of availability if the index being built is on a large collection with many writes during the collection scan. A couple potential solutions:
|
| Comments |
| Comment by Githook User [ 03/Dec/21 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: (cherry picked from commit 9f15b2b1eeb356f4e3d0c6e06302941ec685cc0a) |
| Comment by Githook User [ 02/Dec/21 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: (cherry picked from commit 9f15b2b1eeb356f4e3d0c6e06302941ec685cc0a) |
| Comment by Githook User [ 01/Dec/21 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: (cherry picked from commit 9f15b2b1eeb356f4e3d0c6e06302941ec685cc0a) |
| Comment by Githook User [ 01/Dec/21 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: (cherry picked from commit 9f15b2b1eeb356f4e3d0c6e06302941ec685cc0a) |
| Comment by Bruce Lucas (Inactive) [ 18/Nov/21 ] |
|
harshad.dhavale, I opened backport requests back to 4.2. |
| Comment by Githook User [ 17/Nov/21 ] |
|
Author: {'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}Message: |