[SERVER-44045] allow secondary index builds to start without unlocking the RSTL Created: 16/Oct/19  Updated: 17/Apr/20  Resolved: 17/Apr/20

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Benety Goh Assignee: Louis Williams
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-46989 Index builds should hold RSTL to prev... Closed
Related
related to SERVER-44507 Hybrid index build is able to commit ... Closed
related to SERVER-41033 set ignore_prepare=true throughout an... Closed
is related to SERVER-43415 indexbg_killop_primary.js crashes whe... Closed
is related to SERVER-46704 Two phase index build can violate loc... Closed
is related to SERVER-39451 Add recover to a stable timestamp log... Closed
is related to SERVER-39452 Add rollback via refetch logic for st... Closed
is related to SERVER-41462 do not lock RSTL for uninterruptible ... Closed
is related to SERVER-42824 do not lock RSTL for index build cleanup Closed
is related to SERVER-43837 remove RSTL unlocking logic from Mult... Closed
Sprint: Execution Team 2019-11-04, Execution Team 2020-05-04
Participants:
Linked BF Score: 0

 Description   

One of the planned improvements related to two phase index builds require secondary nodes in a replica set to be able to continue to run index builds on stepping up to primary. This requires the new primary node to hold the RSTL in order to write the commitIndexBuild or abortIndexBuild entries to the oplog.

Index builds on secondary nodes current unlock the RSTL before starting (specifically, the scanning, draining, and completion phases). This logic was introduced in SERVER-41462 to avoid deadlocks with prepared transactions on replica set transitions such as step up/down.

In SERVER-42824, we started unlocking the RSTL on both primary and secondary nodes before cleaning up a failed index build. This logic was recently moved into the IndexBuildsCoordinator in SERVER-43415.

To support two phase index builds, it would be desirable to remove the RSTL unlocking logic.



 Comments   
Comment by Louis Williams [ 17/Apr/20 ]

This is fixed by SERVER-46989

Comment by Suganthi Mani [ 13/Nov/19 ]

benety.goh, SERVER-41462 is to fix a 3 way deadlock if the sequence is something like below.

1) Hybrid index build gets started on secondary for a collection A. And, collection phase of hybrid index build on secondary gets yielded. (Makes all the mongoDB and document level locks to get released - abandons the snapshot).
2) Secondary node steps up and becomes the new primary.
3) Transaction gets stated and inserts some documents in collection A. (This step acquires the collection lock in mode IX)
4) Transaction gets prepared. (After which, it holds the mongoDB locks (database, collection, global) except RSTL lock).
5) Index build started at step #1, gets resumed and moves to drain phase which tries to acquires collection lock in S mode. But, then, it gets blocked on prepared transaction due to collection lock conflict.
5) Now, the node tries to steps down. So, it enqueues the RSTL lock in X mode and blocks behind the index builder (as it holds RSTL lock).
6) Transaction cannot be committed as it blocked behind step down waiting for RSTL lock.

And, the repro script will be the same as the one attached in SERVER-44507.
Note: This 3 way dead lock problem applies even for 2-phase index build.

Comment by Benety Goh [ 18/Oct/19 ]

Was there a reproduction script provided for SERVER-41462? It would help us understand better the underlying issues and have a more productive discussion on alternative solutions.

Comment by Judah Schvimer [ 16/Oct/19 ]

What's the alternative to unlocking the RSTL? I think if we remove the collection S/X locks that index builds acquire then we could. I don't remember us having any alternative to unlocking the RSTL when we added that code.

Comment by Benety Goh [ 16/Oct/19 ]

judah.schvimer, suganthi.mani, any thoughts?

Generated at Thu Feb 08 05:04:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.