Parallelize record store traversal

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Storage Execution
    • Storage Execution 2026-06-08, Storage Execution 2026-06-22, Storage Execution 2026-07-06, Storage Execution 2026-07-20, Storage Execution 2026-08-03, Storage Execution 2026-08-17, Storage Execution 2026-08-31
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This task will make use of ValidateResults objects and their merge functions to run a full record store scan on concurrent threads. 

      It will follow this pattern:

      1. Split the whole record store into slices
        1. use the WiredTiger "random" cursor to quickly find ranges of similar size, note: does not have to be equal
        2. Ensure there are more slices than threads available to allow for "work stealing" to balance that not all records and ranges will be equal
      2. Use the new traverseRecordStore function from SERVER-128372 to generate ValidateResults objects from each slice.  These jobs can be scheduled on the Thread Pool. 
      3. Join all threads
      4. Merge results

      Open Questions:

      What is the impact of the throttle used in the cursor? Will each thread need it's own data throttle? 

      Should we still need to yield? 

      Should we restrict this command to an opt-in from the shell/drivers where the contract will expect that it does not yield and can maximize thread use? 

            Assignee:
            Ayesha Islam
            Reporter:
            Benjamin Pearce
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: