Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16551

tcmalloc ReleaseToCentralCache account > 14% usage with long insert traffic

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.8.0-rc2
    • Component/s: Performance, Storage
    • Labels:
    • Fully Compatible
    • ALL

      during long insert with 64 threads, after about 2500 sec, perf event show tcmalloc's ReleaseToCentralCache is hot, sampled at 14.59% in the following case

      This test is done with wiredtiger, haven;t check mmapv1 yet.

      14.59%    mongod  mongod     [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)

      it seems we can adjust max thread cache size to alleviate this situation. The default size is 32M per thread, did test with 64M, which seems improve the throughput

      and here is perf even flame graph, you can check usage interactively. (please download and open in a new window for better view)

        1. perf.data.22ksec.gz
          12.43 MB
          Rui Zhang
        2. perf.data.22ksec.svg
          626 kB
          Rui Zhang
        3. perf.data.4000sec.gz
          8.45 MB
          Rui Zhang
        4. perf.data.4000sec.svg
          722 kB
          Rui Zhang
        5. perf.data.stable-after-bad3.gz
          2.90 MB
          Rui Zhang
        6. perf-kernel-stable-after-bad3.svg
          403 kB
          Rui Zhang
        7. thread_cache_64mb_vs_32mb.png
          34 kB
          Rui Zhang

            geert.bosch@mongodb.com Geert Bosch
            rui.zhang Rui Zhang (Inactive)
            0 Vote for this issue
            8 Start watching this issue