Implement multithreading of layered table drain.

XMLWordPrintableJSON

    • Type: New Feature
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Layered Tables
    • None
    • Storage Engines - Persistence
    • None
    • None

      Testing indicates that multithreading layered table drain can result in speedups of 2.5-3x in specific scenarios, for this reason I am recommending we implement a simple multithreading solution to enable faster step up times in WiredTiger.

      From my testing around 8 threads in a 16 cpu machine was the magic number, so initially we will hardcode the number of threads to be 8, or half the CPU count if possible.

      To multithread we will:

      • Create a queue of work items (ingest table tree IDs)
      • Spawn a thread group, which will drain the queue and run the relevant ingest drain steps

      This is the simplest version of multithreading, effectively delegating a tree to a thread. There are other implementations possible but testing doesn't necessarily support the increase in complexity yet.

            Assignee:
            Luke Pearson
            Reporter:
            Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: