[SERVER-48002] Do not enforce DataCorruptionDetected assertion when ignoring prepare conflicts Created: 07/May/20 Updated: 29/Oct/23 Resolved: 19/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying, Storage |
| Affects Version/s: | None |
| Fix Version/s: | 4.9.0, 4.4.4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Cheahuychou Mao | Assignee: | Louis Williams |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | qexec-team | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v4.4
|
||||||||
| Sprint: | Execution Team 2020-11-16, Execution Team 2020-11-30 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 34 | ||||||||
| Description |
|
Snapshot isolation cannot be guaranteed for operations that ignore prepare conflicts. This means that two reads of the same record in the same snapshot can return different results. In practice, the assertion added by We cannot enforce the DataCorruptionDetected assertion if we are also ignoring prepare conflicts, or we will continue to get false positives. Original description:
|
| Comments |
| Comment by Githook User [ 17/Dec/20 ] | ||
|
Author: {'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}Message: Snapshot isolation cannot be guaranteed for operations that ignore (cherry picked from commit a01be7ed8e475775ebec57dd6291c3cf5cd33ccf) | ||
| Comment by Louis Williams [ 19/Nov/20 ] | ||
|
Commit: https://github.com/mongodb/mongo/commit/a01be7ed8e475775ebec57dd6291c3cf5cd33ccf | ||
| Comment by Louis Williams [ 16/Nov/20 ] | ||
|
I had to revert my patch because it broke the test designed to exercise this assertion. Most reads ignore prepare conflicts. By making this change, we will completely lose the ability to detect this type of inconsistency for reads, with the exception of multi-doc transactions. This assertion has proven to be somewhat valuable in identifying snapshot isolation bugs in 4.4, but not index consistencies. I am considering the following:
| ||
| Comment by Githook User [ 11/Nov/20 ] | ||
|
Author: {'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}Message: Revert " This reverts commit 523247d096a796c15c911370e622a3614411a25b. | ||
| Comment by Githook User [ 11/Nov/20 ] | ||
|
Author: {'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}Message: Snapshot isolation cannot be guaranteed for operations that ignore | ||
| Comment by Louis Williams [ 03/Nov/20 ] | ||
|
Thanks, justin.seyster for the repro. I think I can see what is happening. A read is ignoring prepare conflicts (default behavior) and is racing with a transaction deleting the document such that it sees the index entry but not the record. Consider the following example with an index on "x":
In general, if an operation ignores prepare conflicts, it cannot guarantee snapshot isolation. This example demonstrates that behavior. If we had not returned an error in this case (behavior before The first solution that comes to mind is to not check this assertion for operations that ignore prepare conflicts. Most reads ignore prepare conflicts, however, so most reads will end up never checking this assertion. | ||
| Comment by Justin Seyster [ 17/Jun/20 ] | ||
|
connie.chen Sure, I just ran this on ToT (ec679c6c), and it failed with DataCorruptionDetected after about an hour of repeating. | ||
| Comment by Connie Chen [ 15/Jun/20 ] | ||
|
justin.seyster would you be able to re-run the torture test on the current code? Storage Engines has made quite a few changes in the last 3 weeks so we want to make sure it's still a problem. | ||
| Comment by Justin Seyster [ 27/May/20 ] | ||
|
Thanks for filing this! After investigating, I don't think that the DataCorruptionDetected uassert() added by From the perspective of the failing shard, it gets a connection from mongoS that starts a multi-document transaction, adds a new document ({x: 1}) via upsert, prepares the transaction, and then commits it. A separate connection from the same monogS queries the collection for the {x: 1} document. The commitTransaction command and the query happen at almost exactly the same time and may be executing concurrently in the mongoD.
Because the commitTransaction and the query are concurrent, it is expected behavior that the query could return 0 or 1 documents, depending on how exactly the two commands are scheduled. However, I do not believe it is expected behavior for the query to observe that the index contains the document {x: 1} but the collection does not (i.e., the collection does not have a record matching the index entry's record id), even if the query happens while while the document is part of a prepared transaction that is in the process of committing. That's what the DataCorruptionDetected uassert() shows is happening, which is why I'm a bit worried about it. This needs some more investigation, but I'm not sure what the next steps are. |