-
Type:
New Feature
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
None
-
Storage Engines - Persistence
-
None
-
None
Testing indicates that multithreading layered table drain can result in speedups of 2.5-3x in specific scenarios, for this reason I am recommending we implement a simple multithreading solution to enable faster step up times in WiredTiger.
From my testing around 8 threads in a 16 cpu machine was the magic number, so initially we will hardcode the number of threads to be 8, or half the CPU count if possible.
To multithread we will:
- Create a queue of work items (ingest table tree IDs)
- Spawn a thread group, which will drain the queue and run the relevant ingest drain steps
This is the simplest version of multithreading, effectively delegating a tree to a thread. There are other implementations possible but testing doesn't necessarily support the increase in complexity yet.
- related to
-
WT-16269 Make 'drain_threads' configurable in test format
-
- Open
-