-
Type: Task
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Labels:
Description
Test setup:Description:
Should mention that sharded transactions can block behind index builds on secondaries.
Engineering Ticket Description:
Original text:
- Multi shard
- collection is sharded with _id: hashed
The evergreen validateCollection hook fails semi-frequently after running the background_index_multikey.js test. Based on initial investigation, it looks like one of the indexes in the secondary is missing a key entry. The primary appears to have passed the validation. Note: test is currently blacklisted.Because transactions yield their locks on secondaries (
SERVER-37199), a concurrent hybrid background index build can conflict in a way that leads to lost writes into building indexes (i.e. corruption) on secondaries.In this example, a background index build on {a: 1} is concurrent with an insert of a document {a: 0} in a transaction while applied on a secondary.
- The background index build converts its X lock to an IX lock while collection scanning. It creates a temporary side-writes table to accept all index key insertions during the build.
- A document {a: 0}is inserted in a transaction and prepared. The key for a: 0 is inserted into the side-writes table as part of the same transaction. When applied on a secondary, it drops its IX locks.
- The background index build takes an X lock, uncontested, and drains the side-writes table. Because the insert into the side-writes table was part of a prepared, but uncommitted transaction, it is invisible to the index builder. The table is then dropped on completion.
- The transaction is finally committed, but its side-write is committed to a now-deleted table. The inserted key is now lost forever and the resulting index is corrupted.
On a primary, our locks prevent this from happening, but because an index build can complete while a prepared transaction is active, we can lose writes into building indexes.
edit: louis.williams
Scope of changes
Impact to Other Docs
MVP (Work and Date)
Resources (Scope or Design Docs, Invision, etc.)
- documents
-
SERVER-38588 Hybrid index builds do not work when applied concurrently with prepared transactions on secondaries
- Closed
- related to
-
DOCS-12523 4.2: transactions supported on sharded clusters (part 1 March 14)
- Closed