Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7182

Long latencies for some requests with large HS and heavy update workload

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 8
    • Storage - Ra 2021-06-28, Storage - Ra 2021-07-12, Storage - Ra 2021-07-26

      Some requests show latencies of more than 10 seconds when running a long workload (~5 million updates) with the stable timestamp pinned.

      I reran the test workload from PM-1844 on 4.4.4-rc0 to see how it performs with the fix for WT-6924. The test workload has a long "grow" phase, which performs a lot of updates (with a few inserts and deletes) designed to grow the WT history store.  Before the fix for WT-6924, this phase cause the WT cache to vastly exceed its configured size limits which also caused many updates to have high latencies.  

      In this test, the WT cache adheres to its configured size, and update throughput looks more consistent through much of the test.  But by the end of the test we are still seeing much lower performance and some individual operations (~9000) have latencies > 1 second.

      I'll attach data.tgz with the FTDC data and mongod log file.

        1. 07_v2_flame_02.png
          07_v2_flame_02.png
          369 kB
        2. b6_7_hits_v2_new.png
          b6_7_hits_v2_new.png
          85 kB
        3. Batch6_7.png
          Batch6_7.png
          2.33 MB
        4. Batch7_02.png
          Batch7_02.png
          4.19 MB
        5. data.tgz
          28.36 MB
        6. evergreen_vol.png
          evergreen_vol.png
          209 kB
        7. file_level_stats_hs.png
          file_level_stats_hs.png
          103 kB
        8. file_level_stats_idx34.png
          file_level_stats_idx34.png
          103 kB
        9. file_level_stats_idx35.png
          file_level_stats_idx35.png
          100 kB
        10. file_level_stats_idx36.png
          file_level_stats_idx36.png
          96 kB
        11. file_level_stats_idx41.png
          file_level_stats_idx41.png
          87 kB
        12. file_level_stats_idx43.png
          file_level_stats_idx43.png
          88 kB
        13. fs_chedkpoint.png
          fs_chedkpoint.png
          186 kB
        14. heatmap_latency_a_b.png
          heatmap_latency_a_b.png
          359 kB
        15. hs_flame_graph_evict.png
          hs_flame_graph_evict.png
          358 kB
        16. hs_noaws.png
          hs_noaws.png
          165 kB
        17. io_flame.png
          io_flame.png
          419 kB
        18. latency flame.png
          latency flame.png
          261 kB
        19. Screen Shot 2021-02-09 at 10.23.48 PM.png
          Screen Shot 2021-02-09 at 10.23.48 PM.png
          195 kB
        20. Screen Shot 2021-07-15 at 12.53.56 pm.png
          Screen Shot 2021-07-15 at 12.53.56 pm.png
          135 kB
        21. t2_hs_overview.png
          t2_hs_overview.png
          270 kB
        22. ts_hs_eviction.png
          ts_hs_eviction.png
          173 kB

            Assignee:
            alison.felizzi@mongodb.com Alison Felizzi (Inactive)
            Reporter:
            keith.smith@mongodb.com Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: