Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7619

Add a new optimization to skip pages in cursor traversal when all entries on the page are deleted



    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT10.0.1, 5.1.0, 4.4.8, 5.0.2
    • Component/s: None
    • Labels:
    • Story Points:
    • Sprint:
      Storage - Ra 2021-06-14, Storage - Ra 2021-06-28, Storage - Ra 2021-07-12, Storage - Ra 2021-07-26


      This is a break-off ticket fromĀ WT-6796 as we want to understand why delete throughout decreases as some of the remove-only benchmarks progress.

      There are few things to consider here:

      • As discussed inĀ WT-6796, the reduced delete operation doesn't seem to correlate with history store file size.
      • The delete throughout is decreasing as we keep deleting random documents from the test collections without replacing them.
      • The benchmark is specifically testing system performance with long running transactions. From the plot above we can notice that the stat "range of timestamps currently pinned" is growing to 1 hour.
      • Deleting records in WT essentially entails adding a stop timestamp the records. The actual record in not removed until the stop time becomes globally visible.
      • We need to gather more information about the benchmark. For example, how cursors are being used to delete the data and if there any indexes on the test collection.

      The aim of this ticket is to find the root cause behind the delete performance degradation and create tickets for possible improvements.


        1. FinalChangesOneHourPhase.png
          348 kB
          Haseeb Bokhari
        2. full-range.png
          48 kB
          James O'Leary
        3. full-range-benchmark.png
          47 kB
          James O'Leary
        4. hist-collection0.png
          61 kB
          James O'Leary
        5. hist-half-range.png
          24 kB
          James O'Leary
        6. image-2021-05-31-16-55-40-718.png
          593 kB
          Haseeb Bokhari
        7. image-2021-06-10-15-00-55-476.png
          425 kB
          Haseeb Bokhari
        8. image-2021-06-11-09-58-12-690.png
          342 kB
          Haseeb Bokhari
        9. image-2021-06-11-09-59-21-306.png
          483 kB
          Haseeb Bokhari
        10. image-2021-06-11-10-24-38-903.png
          566 kB
          Haseeb Bokhari
        11. image-2021-06-11-10-31-36-408.png
          608 kB
          Haseeb Bokhari
        12. image-2021-06-11-10-35-23-972.png
          681 kB
          Haseeb Bokhari
        13. image-2021-06-15-14-02-10-227.png
          373 kB
          Haseeb Bokhari
        14. image-2021-06-15-14-03-19-961.png
          377 kB
          Haseeb Bokhari
        15. image-2021-06-15-14-50-04-635.png
          410 kB
          Haseeb Bokhari
        16. image-2021-06-23-11-06-43-745.png
          453 kB
          Haseeb Bokhari
        17. image-2021-07-01-17-01-08-037.png
          163 kB
          Haseeb Bokhari
        18. image-2021-07-02-21-45-26-020.png
          346 kB
          Haseeb Bokhari
        19. LatestOptimization_10min_window.png
          366 kB
          Haseeb Bokhari
        20. optimization-partial.png
          271 kB
          Haseeb Bokhari
        21. RemoveGtc.png
          67 kB
          James O'Leary
        22. RemoveLtc.png
          60 kB
          James O'Leary
        23. skipped-by-cursor-next-new.png
          383 kB
          Haseeb Bokhari
        24. slow-delete.png
          391 kB
          Haseeb Bokhari
        25. workgen-new-benchmark.png
          275 kB
          Haseeb Bokhari

          Issue Links



              haseeb.bokhari Haseeb Bokhari
              haseeb.bokhari Haseeb Bokhari
              0 Vote for this issue
              13 Start watching this issue