[SERVER-74953] Explore avoiding stepdowns during the early phases of index build setup Created: 16/Mar/23  Updated: 04/Jul/23  Resolved: 15/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Yujin Kang Park Assignee: Yujin Kang Park
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 0001-SERVER-74953-reproducer.patch    
Issue Links:
Depends
depends on SERVER-70127 Default system operations to be killa... Closed
Related
related to SERVER-78143 Complete TODO listed in SERVER-74953 Closed
is related to SERVER-44250 startIndexBuild oplog write and threa... Closed
is related to SERVER-77025 Command application conflicting with ... Closed
is related to SERVER-75059 Explore improvements on index build c... Backlog
Sprint: Execution Team 2023-03-20, Execution Team 2023-04-03, Execution Team 2023-04-17, Execution Team 2023-05-01, Execution Team 2023-05-29, Execution Team 2023-06-12, Execution EMEA Team 2023-06-26
Participants:

 Description   

Explore avoiding stepdowns during the early phases of index build setup, until the build reaches kPostSetup.

Original description

An index build can register and get stuck trying to acquire a ticket to actually do the setup (and replicate startIndexBuild). So it is possible for the index build to be registered, when a stepdown happens, which does not kill the builders opCtx, and the build is still stuck after going into steady state replication as secondary. The new primary drops the collection, and on applying the drop in the secondary, we assert that there are no builds in progress... but there is one which is registered because the old primary has not yet advanced to a point where it checks if it is still primary.

Being blocked on ticket acquisition is not a necessity for this to manifest, as a sufficiently slow machine could theoretically suffer from the same issue.



 Comments   
Comment by Githook User [ 04/Jul/23 ]

Author:

{'name': 'Yu Jin Kang Park', 'email': 'yujin.kang@mongodb.com', 'username': 'ykangpark'}

Message: SERVER-78143 Remove TODO SERVER-74953
Branch: master
https://github.com/mongodb/mongo/commit/a897690f1a192921fa8ddfdd002ed2513af14275

Comment by Yujin Kang Park [ 15/Jun/23 ]

 SERVER-77025 fixed the bug described in this ticket by reverting to a previous behaviour. Avoiding stepdowns during setup or delaying registering the index build until setup is complete is not trivial. This would be better addressed more holistically in a re-design of index builds. Since the bug is fixed, closing as "Won't Fix".

Comment by Yujin Kang Park [ 25/May/23 ]

Linking SERVER-44250 which explains why it was necessary for the index build setup (and 'startIndexBuild' replication) to happen within the index builder thread. 

Comment by Josef Ahmad [ 11/May/23 ]

Proposing resolving this bug as part of SERVER-77025, which reverts the code that introduces the bug in the first place. This is going to be less risky to backport to 7.0 than the longer-term improvement described in this ticket SERVER-74953.

Comment by Josef Ahmad [ 11/May/23 ]

I reproduced the issue on 6.1.0 and proved it does not reproduce by reverting SERVER-61481. This bug predates the "Gracefully Handle Index Build Errors" project.

Comment by Josef Ahmad [ 05/May/23 ]

This bug may have been around since 6.1, due to SERVER-61481. That ticket removed the ability for replication to wait for an index build to complete before applying a DDL operation to the namespace. With that logic removed, the race described in this ticket results in an invariant failure. cc: gregory.noma@mongodb.com pavithra.vetriselvan@mongodb.com 

In any case, I believe the index build setup phase should ideally be atomic. Preventing an election from interleaving with the setup avoids an undesirable state where a new primary isn't aware that an index build has already started setting up.

(I just attached yujin.kang@mongodb.com's reproducer)

Generated at Thu Feb 08 06:28:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.