-
Type: Improvement
-
Resolution: Fixed
-
Priority: Critical - P2
-
Affects Version/s: None
-
Component/s: None
-
(copied to CRM)
-
5
-
Storage Engines - 2022-10-31, Storage Engines - 2022-11-14
-
v6.1
Summary
WiredTiger has stopped actively cleaning up obsolete deleted leaf pages from tables (collections and indexes). This is because internal pages that reference deleted children, are no longer considered for reconciliation - which is the process that removes obsolete deleted pages.
Obsolete here means that no reader in the system could want to see the content of the page.
Deleted pages are artifacts of an optimization associated with the WT_SESSION::truncate implementation of WiredTiger, which marks entire on-disk pages as deleted, rather than deleting every individual entry on that page.
Motivation
- Does this affect any team outside of WT?
Yes - the MongoDB Oplog uses the WiredTiger truncate operation to maintain it's size. In certain situations, obsolete children are never being cleaned up.
This results in increased disk space usage for MongoDB oplog.
- How likely is it that this use case or problem will occur?
The problem occurs if an internal page is reconciled (written back from cache to disk) after a truncate operation has visited that part of the tree, while there are other transactions in the system that (for reasons of snapshot isolation) would need to see the state of the database prior to the truncate operation.
There are two potential causes for such a reconciliation:
- Checkpoints write internal pages back to disk - in MongoDB checkpoint runs once per minute.
- In a system with a lot of cache pressure, internal pages may be evicted to make space for other content in the cache.
Having this pattern happen regularly is much more likely for applications that have a lot of active concurrent transactions, or have long running multi-document transactions.
- If the problem does occur, what are the consequences and how severe are they?
The oplog data file will continuously grow, which can lead to disk space availability issues.
- Is this issue urgent?
Yes - a MongoDB customer encountered this issue, so it is a very high priority.