[SERVER-65106] Don't perform eviction after prepared transaction commits or rolls-back Created: 31/Mar/22  Updated: 29/Oct/23  Resolved: 27/Feb/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.0.0-rc0

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-9040 Configuration for transaction commit ... Closed
Related
related to SERVER-64613 Periodically wake up when waiting for... Closed
related to WT-8975 Eviction is triggered with prepared t... Closed
is related to SERVER-71674 Yield handlers not handling WCE throw... Blocked
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2023-03-06
Participants:
Linked BF Score: 10

 Description   

This is an alternative solution to both SERVER-64613 and WT-8975.

When MongoDB encounters a prepare conflict in WiredTiger, we wait until an active MongoDB WriteUnitOfWork commits or aborts.

There is a deadlock caused by this approach that can be described as follows:

  • A prepared transaction has updated a document.
  • A concurrent operation tries to read that document, but it hits a prepare conflict. It waits until the prepared transactions commits or aborts (via a notification). While it waits, its cursor remains positioned, preventing the page from being evicted.
  • The prepared transaction aborts or commits, but after the transition rolls-back, it is recruited for bonus eviction. It cannot evict anything because the page is pinned by the reader. So even though the transaction has been successfully rolled back in the storage engine, cannot make progress to signal the reader to continue. The reader is waiting for a notification, which would unpin the page.

This is only a bug in a system with one prepared transaction, since any transactions committing or aborting would wake the reader.

We want to break this deadlock in MongoDB by requesting to WiredTiger that the operation not perform eviction after it rolls back or commits.



 Comments   
Comment by Louis Williams [ 27/Feb/23 ]

This implements the solution proposed in SERVER-64613 to periodically wake up blocked operations.

Even though this problem exists on older branches, I am choosing not to request backports because this problem has only ever been reproduced in abnormal WT cache configurations.

Comment by Githook User [ 24/Feb/23 ]

Author:

{'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}

Message: SERVER-65106 Periodically wake to check if prepared transaction has ended
Branch: master
https://github.com/mongodb/mongo/commit/d5f4a148778c761dbee720106786c8290fec428b

Comment by Connie Chen [ 13/Feb/23 ]

We think the simplest solution is to periodically wake up and see if a transaction has been committed

Comment by Louis Williams [ 13/Apr/22 ]

keith.smith@mongodb.com, yes, the reader has an active transaction. While I think the answer is "yes", we could reset the cursor while waiting, it would require some significant changes to our code. We instrument waiting for prepare conflicts in almost every cursor API call we make to WiredTiger. We would probably need to expand the scope of each retry loop to include the part that positions the cursor. And I imagine we would have to carefully consider each loop to ensure each sequence is safely retryable.

Comment by Keith Smith [ 12/Apr/22 ]

Step two of the scenario on the description says:

A concurrent operation tries to read that document, but it hits a prepare conflict. It waits until the prepared transactions commits or aborts (via a notification). While it waits, its cursor remains positioned, preventing the page from being evicted.

Just to clarify, does the reader here have an active transaction while it is waiting?  If so, is there other transaction state that may be in play here (i.e., updates from previous operations)? If the read cursor is really the only WT resource the reader has, could it just reset the cursor before sleeping?  

Comment by Chenhao Qu [ 31/Mar/22 ]

louis.williams I don't think there is a way to do that at the moment. But I think we can add a configuration in transaction commit to achieve that.

Comment by Louis Williams [ 31/Mar/22 ]

chenhao.qu, agorrod, etienne.petrel: Is there any way to ask WiredTiger that a transaction not perform eviction when it commits? We currently use "operation_timeout_ms" in certain cases of rollback, but I don’t believe this works for commit_transaction. I ask because WT-8975 was closed, but we don’t have a way to achieve the same behavior in MongoDB. We are not thrilled with SERVER-64613 as a solution (periodically waking up) because we believe it masks potential bugs and is a workaround rather than a long-term solution.

Generated at Thu Feb 08 06:01:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.