Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7182

Long latencies for some requests with large HS and heavy update workload

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 8
    • Storage - Ra 2021-06-28, Storage - Ra 2021-07-12, Storage - Ra 2021-07-26

      Some requests show latencies of more than 10 seconds when running a long workload (~5 million updates) with the stable timestamp pinned.

      I reran the test workload from PM-1844 on 4.4.4-rc0 to see how it performs with the fix for WT-6924. The test workload has a long "grow" phase, which performs a lot of updates (with a few inserts and deletes) designed to grow the WT history store.  Before the fix for WT-6924, this phase cause the WT cache to vastly exceed its configured size limits which also caused many updates to have high latencies.  

      In this test, the WT cache adheres to its configured size, and update throughput looks more consistent through much of the test.  But by the end of the test we are still seeing much lower performance and some individual operations (~9000) have latencies > 1 second.

      I'll attach data.tgz with the FTDC data and mongod log file.

        1. data.tgz
          28.36 MB
        2. Screen Shot 2021-02-09 at 10.23.48 PM.png
          Screen Shot 2021-02-09 at 10.23.48 PM.png
          195 kB
        3. t2_hs_overview.png
          t2_hs_overview.png
          270 kB
        4. ts_hs_eviction.png
          ts_hs_eviction.png
          173 kB
        5. hs_flame_graph_evict.png
          hs_flame_graph_evict.png
          358 kB
        6. fs_chedkpoint.png
          fs_chedkpoint.png
          186 kB
        7. b6_7_hits_v2_new.png
          b6_7_hits_v2_new.png
          85 kB
        8. heatmap_latency_a_b.png
          heatmap_latency_a_b.png
          359 kB
        9. latency flame.png
          latency flame.png
          261 kB
        10. io_flame.png
          io_flame.png
          419 kB
        11. Batch6_7.png
          Batch6_7.png
          2.33 MB
        12. Batch7_02.png
          Batch7_02.png
          4.19 MB
        13. 07_v2_flame_02.png
          07_v2_flame_02.png
          369 kB
        14. file_level_stats_hs.png
          file_level_stats_hs.png
          103 kB
        15. file_level_stats_idx34.png
          file_level_stats_idx34.png
          103 kB
        16. file_level_stats_idx35.png
          file_level_stats_idx35.png
          100 kB
        17. file_level_stats_idx36.png
          file_level_stats_idx36.png
          96 kB
        18. file_level_stats_idx41.png
          file_level_stats_idx41.png
          87 kB
        19. file_level_stats_idx43.png
          file_level_stats_idx43.png
          88 kB
        20. hs_noaws.png
          hs_noaws.png
          165 kB
        21. evergreen_vol.png
          evergreen_vol.png
          209 kB
        22. Screen Shot 2021-07-15 at 12.53.56 pm.png
          Screen Shot 2021-07-15 at 12.53.56 pm.png
          135 kB

            Assignee:
            alison.felizzi@mongodb.com Alison Felizzi (Inactive)
            Reporter:
            keith.smith@mongodb.com Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: