Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7182

Long latencies for some requests with large HS and heavy update workload

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
    • 8
    • Storage - Ra 2021-06-28, Storage - Ra 2021-07-12, Storage - Ra 2021-07-26

      Some requests show latencies of more than 10 seconds when running a long workload (~5 million updates) with the stable timestamp pinned.

      I reran the test workload from PM-1844 on 4.4.4-rc0 to see how it performs with the fix for WT-6924. The test workload has a long "grow" phase, which performs a lot of updates (with a few inserts and deletes) designed to grow the WT history store.  Before the fix for WT-6924, this phase cause the WT cache to vastly exceed its configured size limits which also caused many updates to have high latencies.  

      In this test, the WT cache adheres to its configured size, and update throughput looks more consistent through much of the test.  But by the end of the test we are still seeing much lower performance and some individual operations (~9000) have latencies > 1 second.

      I'll attach data.tgz with the FTDC data and mongod log file.

        1. 07_v2_flame_02.png
          369 kB
          Alison Felizzi
        2. b6_7_hits_v2_new.png
          85 kB
          Alison Felizzi
        3. Batch6_7.png
          2.33 MB
          Alison Felizzi
        4. Batch7_02.png
          4.19 MB
          Alison Felizzi
        5. data.tgz
          28.36 MB
          Keith Smith
        6. evergreen_vol.png
          209 kB
          Alison Felizzi
        7. file_level_stats_hs.png
          103 kB
          Alison Felizzi
        8. file_level_stats_idx34.png
          103 kB
          Alison Felizzi
        9. file_level_stats_idx35.png
          100 kB
          Alison Felizzi
        10. file_level_stats_idx36.png
          96 kB
          Alison Felizzi
        11. file_level_stats_idx41.png
          87 kB
          Alison Felizzi
        12. file_level_stats_idx43.png
          88 kB
          Alison Felizzi
        13. fs_chedkpoint.png
          186 kB
          Alison Felizzi
        14. heatmap_latency_a_b.png
          359 kB
          Alison Felizzi
        15. hs_flame_graph_evict.png
          358 kB
          Alison Felizzi
        16. hs_noaws.png
          165 kB
          Alison Felizzi
        17. io_flame.png
          419 kB
          Alison Felizzi
        18. latency flame.png
          261 kB
          Alison Felizzi
        19. Screen Shot 2021-02-09 at 10.23.48 PM.png
          195 kB
          Keith Smith
        20. Screen Shot 2021-07-15 at 12.53.56 pm.png
          135 kB
          Alison Felizzi
        21. t2_hs_overview.png
          270 kB
          Alison Felizzi
        22. ts_hs_eviction.png
          173 kB
          Alison Felizzi

            Assignee:
            alison.felizzi@mongodb.com Alison Felizzi (Inactive)
            Reporter:
            keith.smith@mongodb.com Keith Smith
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: