One of the challenges for WiredTiger, as a lock-free btree implementation is splitting hot pages. A hot page being defined as a page that has multiple threads accessing and updating it concurrently.

There are a set of internal trade-offs related to hot page management, including:

• Generally, we need to ensure mechanisms aimed at managing hot pages don't introduce performance regressions when accessing non-hot pages. i.e: identifying hot pages and treating them specially is essential.
• Allowing hot pages to grow too large can mean that the cost of splitting (and reconciling) will be high. When exclusive access is gained, it can lead to the operation waiting on the page and needing to wait a long time.
• Sometimes checkpoints create blocks of time when a split is not possible. In that case, if a page is allowed to grow arbitrarily large - the above latency becomes an issue. It can also lead to a "robbing Peter to pay Paul" pattern - where applications have high throughput during checkpoint (there is no attempt to gain exclusive access, so updates are made aggressively). Once the checkpoint completes - threads enter a phase of coordinated backoff which slows throughput down. Once exclusive access is gained, reconciliation takes a long time (which appears like a stall for operations working on that page).

The workloads should test:

• Only update operations.
• Only insert operations to hot sections.
• Single hot pages.
• Multiple hot pages spread across a larger dataset.
• Combinations of insert/update and read threads at different levels of concurrency.
• Some extreme concurrency tests with many threads contending a page.

The tests need to be careful not to make the job of exclusive access impossible for WiredTiger - cursors/threads need to acquire/release references to pages regularly.

It would be useful to create a set of workloads that create hot pages in various different forms, and adds tracking for overall throughput, throughput stability and P95, P99 and maximum latency.

Once we have those performance tests we can more confidently make changes to how we manage hot pages, and gaining exclusive access to them.

alexander.gorrod@mongodb.com has 1000 bonusly on offer for anyone who creates this set of tests..