[SERVER-45382] indexbg_restart_secondary.js can race processing a commitIndexBuild oplog entry and interrupting an index build due to shutdown Created: 06/Jan/20 Updated: 29/Oct/23 Resolved: 22/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 4.3.3 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Louis Williams | Assignee: | Louis Williams |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Sprint: | Execution Team 2020-01-27 | ||||||||||||
| Participants: | |||||||||||||
| Linked BF Score: | 18 | ||||||||||||
| Description |
|
When secondaries receive exceptions during index builds, they rethrow immediately. This results in the index build also getting torn down immediately, because we call abortIndexBuildWithoutCleanup() while tearing down the index build. Once the index build thread tears down, it unregisters itself. If the oplog applier is concurrently processing of the commitIndexBuild oplog entry, the call to abortIndexBuild will throw a NoSuchKey error here. We ignore NoSuchKey, so the commitIndexBuild gets processed and the stable timestamp is allowed to advance. Upon restarting, the unfinished index build will never be completed because the "commitIndexBuild" has already been processed without successfully committing the index. |
| Comments |
| Comment by Githook User [ 18/Jan/20 ] |
|
Author: {'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}Message: |
| Comment by Louis Williams [ 09/Jan/20 ] |
|
This is actually just a test issue. In actual shutdown, we stop the replication coodinator (and batch applier threads) before we kill all operations: https://github.com/mongodb/mongo/blob/9cc6478f77febe70b10742e0e92514942936bef3/src/mongo/db/db.cpp#L985-L997. |