[SERVER-45382] indexbg_restart_secondary.js can race processing a commitIndexBuild oplog entry and interrupting an index build due to shutdown Created: 06/Jan/20  Updated: 29/Oct/23  Resolved: 22/Jan/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-44953 Secondaries should restart index buil... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2020-01-27
Participants:
Linked BF Score: 18

 Description   

When secondaries receive exceptions during index builds, they rethrow immediately. This results in the index build also getting torn down immediately, because we call abortIndexBuildWithoutCleanup() while tearing down the index build.

Once the index build thread tears down, it unregisters itself. If the oplog applier is concurrently processing of the commitIndexBuild oplog entry, the call to abortIndexBuild will throw a NoSuchKey error here. We ignore NoSuchKey, so the commitIndexBuild gets processed and the stable timestamp is allowed to advance.

Upon restarting, the unfinished index build will never be completed because the "commitIndexBuild" has already been processed without successfully committing the index.



 Comments   
Comment by Githook User [ 18/Jan/20 ]

Author:

{'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}

Message: SERVER-45382 indexbg_restart_secondary.js should shutdown secondary normally
Branch: master
https://github.com/mongodb/mongo/commit/044e97ded5fe32ed79910d58c45f5339cb7da46c

Comment by Louis Williams [ 09/Jan/20 ]

This is actually just a test issue. In actual shutdown, we stop the replication coodinator (and batch applier threads) before we kill all operations: https://github.com/mongodb/mongo/blob/9cc6478f77febe70b10742e0e92514942936bef3/src/mongo/db/db.cpp#L985-L997.

Generated at Thu Feb 08 05:08:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.