When an uncommitted transaction becomes so large that it must be rolled back, WiredTiger returns a WT_ROLLBACK error code. MongoDB propagates this to users as a WriteConflict. For MongoDB users of multi-document transactions, this is confusing and hard for us to explain because the transaction was actually using too much memory.
EDIT: Additional use of this feature is to permit MongoDB code logic to handle this type of rollback differently from others. In particular, we want to add some complexity to the retry-mechanism based on the reason for the rollback.
We should consider either using a different error code or updating the WiredTiger API to include some context in the error message. I'm interested in the "oldest pinned transaction ID rolled back for eviction", which I believe is the event that triggers a rollback when the dirty cache limit is hit.