[SERVER-40820] Jstestfuzz suite to introduce spurious write conflicts Created: 25/Apr/19  Updated: 29/Oct/23  Resolved: 26/Feb/21

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.4.7, 5.0.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Louis Williams Assignee: Bynn Lee
Resolution: Fixed Votes: 0
Labels: newgrad
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-51593 Index build bulk builder does not han... Backlog
depends on SERVER-51526 Hybrid index builds can miss writes a... Closed
depends on SERVER-51656 CollectionBulkLoader does not handle ... Closed
Related
is related to SERVER-21332 Add FSM runners that always enable th... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.0, v4.4
Sprint: Execution Team 2021-02-22, Execution Team 2021-03-08
Participants:

 Description   

In an attempt to reproduce some failures, I noticed we have remarkably little coverage of handling spurious WriteConflicts, especially in operations where they are normally unexpected. While most of our code is designed to handle such cases, I think we should exercise these code paths more often, since WiredTiger can return WT_ROLLBACK at any time.

I would suggest enabling the WTWriteConflictException and WTWriteConflictExceptionForReads fail points to a very low frequency (e.g. < 1%) so that most operations can make some progress.



 Comments   
Comment by Githook User [ 10/May/21 ]

Author:

{'name': 'Bynn Lee', 'email': 'bynn.lee@mongodb.com', 'username': 'bynn'}

Message: SERVER-40820 Index builders to support write conflicts

(cherry picked from commit 614dbf274a3939359d9e351ce74d1d281dc12de3)
Branch: v4.4
https://github.com/mongodb/mongo/commit/17356e559751ee280733158022a6c1ce755a741b

Comment by Louis Williams [ 18/Mar/21 ]

We should backport the first commit https://github.com/mongodb/mongo/commit/614dbf274a3939359d9e351ce74d1d281dc12de3 to fix certain failures that could happen in 4.4

Comment by Githook User [ 26/Feb/21 ]

Author:

{'name': 'Bynn Lee', 'email': 'bynn.lee@mongodb.com', 'username': 'bynn'}

Message: SERVER-40820 Jstestfuzz suite to introduce spurious write conflicts
Branch: master
https://github.com/mongodb/mongo/commit/43224bbbba4f145093eec87d8ba6554a5c541118

Comment by Githook User [ 26/Feb/21 ]

Author:

{'name': 'Bynn Lee', 'email': 'bynn.lee@mongodb.com', 'username': 'bynn'}

Message: SERVER-40820 Index builders to support write conflicts
Branch: master
https://github.com/mongodb/mongo/commit/614dbf274a3939359d9e351ce74d1d281dc12de3

Comment by Louis Williams [ 15/Oct/20 ]

Because of the impact of SERVER-51593, we should make sure to add WT_OP_CHECK s around inserts into the bulk loader so we can simulate throwing write conflicts at that point.

Comment by Louis Williams [ 15/Oct/20 ]

From SERVER-43053 that I closed as a dupe:

SERVER-42330 blacklists the WriteConflict-inducing workloads it adds from concurrency_simultaneous*.yml, since they presently fail because of the WriteConflict exception handling (or lack thereof) of various commands. These workloads should be un-blacklisted as part of SERVER-43053.

Comment by Max Hirschhorn [ 25/Apr/19 ]

I chatted with Louis in-person about this ticket. It sounds like setting the WTWriteConflictException and WTWriteConflictExceptionForRead failpoints had been a useful tactic when debugging some concurrency test failures. They did manifest as crashes (in renameCollection) and test timeouts so it might be the case the fuzzer (or at least the concurrent version of it) would also trigger these cases. My initial reaction to this ticket was whether changing these knobs would be more in a new concurrency test suite rather than a new fuzzer test suite. I think it'd be good to clarify the kinds of bugs we'd expect to find by having the two failpoints enabled.

Generated at Thu Feb 08 04:56:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.