[DOCS-12538] Docs for SERVER-38588: Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries Created: 11/Mar/19  Updated: 13/Nov/23  Resolved: 26/Dec/19

Status: Closed
Project: Documentation
Component/s: manual, Server
Affects Version/s: None
Fix Version/s: 4.1.9, Server_Docs_20231030, Server_Docs_20231106, Server_Docs_20231105, Server_Docs_20231113

Type: Task Priority: Major - P3
Reporter: Kay Kim (Inactive) Assignee: Ravind Kumar (Inactive)
Resolution: Done Votes: 0
Labels: docs-txns, open_todo_in_code
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
documents SERVER-38588 Hybrid index builds do not work when ... Closed
Related
related to DOCS-12523 4.2: transactions supported on sharde... Closed
Participants:
Days since reply: 4 years, 6 weeks, 6 days ago
Epic Link: DOCS: 4.2 Server/Tools

 Description   

Description

Description:

Should mention that sharded transactions can block behind index builds on secondaries.

Engineering Ticket Description:

Original text:

Test setup: 

  • Multi shard 
  • collection is sharded with _id: hashed 
     
    The evergreen validateCollection hook fails semi-frequently after running the background_index_multikey.js test. Based on initial investigation, it looks like one of the indexes in the secondary is missing a key entry. The primary appears to have passed the validation. Note: test is currently blacklisted.

    Because transactions yield their locks on secondaries (SERVER-37199), a concurrent hybrid background index build can conflict in a way that leads to lost writes into building indexes (i.e. corruption) on secondaries.

    In this example, a background index build on {a: 1} is concurrent with an insert of a document {a: 0} in a transaction while applied on a secondary.

    • The background index build converts its X lock to an IX lock while collection scanning. It creates a temporary side-writes table to accept all index key insertions during the build.
    • A document {a: 0}is inserted in a transaction and prepared. The key for a: 0 is inserted into the side-writes table as part of the same transaction. When applied on a secondary, it drops its IX locks.
    • The background index build takes an X lock, uncontested, and drains the side-writes table. Because the insert into the side-writes table was part of a prepared, but uncommitted transaction, it is invisible to the index builder. The table is then dropped on completion.
    • The transaction is finally committed, but its side-write is committed to a now-deleted table. The inserted key is now lost forever and the resulting index is corrupted.

    On a primary, our locks prevent this from happening, but because an index build can complete while a prepared transaction is active, we can lose writes into building indexes.

    edit: louis.williams

Scope of changes

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)



 Comments   
Comment by Githook User [ 26/Dec/19 ]

Author:

{'name': 'rk-mongo', 'email': 'ravind.kumar@mongodb.com'}

Message: DOCS-12538: Secondary Index Builds block Transactions
Branch: master
https://github.com/mongodb/docs/commit/53ce4c22fb9072ab8ec1cfe3b0881dda21424506

Comment by Ravind Kumar (Inactive) [ 12/Jun/19 ]

Partially completed as part of DOCSP-4986

Docs

To be more complete, we can add a note to the production considerations (sharded clusters) section of the docs.

Generated at Thu Feb 08 08:05:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.