Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-2131

Improve Linux transparent hugepages details in production notes



    • Task
    • Status: Closed
    • Critical - P2
    • Resolution: Fixed
    • None
    • None
    • manual
    • None
    • 4
    • Docs Sprint 17 - Ending 4/3
    • true


      The Linux Recommended Configuration section of the MongoDB production notes states

      Do not use hugepages virtual memory pages as MongoDB performs better with normal virtual memory pages.

      There are a few issues with this.

      1. It conflates "hugepages" and "transparent hugepages" (THP), which are related but not the same thing.
        1. Hugepages must be preallocated and require application-level support (which MongoDB does not have).
        2. Transparent hugepages (THP) are an operating system feature to automatically and transparently (to the application) use hugepages when possible.
      2. No information is given as to how users can:
        1. Which Linux distributions (and versions) have THP enabled by default.
        2. How users can check whether or not THP is enabled.
        3. How users can investigate potential performance problems arising from THP.
        4. How users can disable THP (temporarily and/or permanently).

      Regarding the first point, it would be sufficient to adjust the sentence to read "Ensure that transparent hugepages are disabled, as MongoDB performs better with normal virtual memory pages" or some-such.

      I believe that THP is only enabled by default in RHEL6 and SLES11 (and derivatives), and is disabled otherwise, but this might not be the case. If it is, this issue's relevance can be expected to increase over time as existing RHEL5 based systems are progressively upgraded to RHEL6.

      To check if THP is enabled, users need to check the contents of /sys/kernel/mm/transparent_hugepage/enabled or /sys/kernel/mm/redhat_transparent_hugepage/enabled on Redhat-derived systems (ie. RHEL, CentOS, Amazon, etc). Some systems may have /sys/kernel/mm/transparent_hugepage as a symlink to /sys/kernel/mm/redhat_transparent_hugepage. Users could potentially be instructed to run

      cat /sys/kernel/mm/*transparent_hugepage/enabled

      to determine this in a Linux distro agnostic way. If neither of these files are present, then THP support is not compiled into the Linux kernel, and so is as good as disabled.

      If THP is enabled, the output will look like:

      [always] never

      or on some systems:

      [always] madvise never

      If THP is disabled, the output will look like one of:

      always [never]

      always madvise [never]

      always [madvise] never

      To check possible performance problems arising from THP, users can check the khugepaged kernel thread process, as well as the contents of the other files in /sys/kernel/mm/*transparent_hugepage. In particular, any of the following are red flags:

      • enabled set to always
      • defrag set to always
      • khugepaged/defrag set to yes or 1
      • High values in any of khugepaged/full_scans, khugepaged/pages_collapsed or khugepaged/pages_to_scan
      • High cumulative cpu time for the khugepaged process
      • High cpu% for the khugepaged process
      • High or increasing value of AnonHugePages in /proc/meminfo
      • High or increasing value of thp_fault_alloc (or any other thp_* value) in /proc/vmstat

      To disable THP temporarily (eg. to assess any possible impact), users can run as root:

      for i in /sys/kernel/mm/*transparent_hugepage/enabled; do echo never > $i; done

      or equivalently as a regular user:

      echo never | sudo tee /sys/kernel/mm/*transparent_hugepage/enabled

      This will stop khugepaged from performing any actions. Restarting mongod is not required for the new THP settings to take effect.

      When disabling THP, any previously allocated transparent hugepages (reported by AnonHugePages in /proc/meminfo) will not be broken up. In most situations, this is unlikely to be a problem, but if it is suspected, they can be cleared by rebooting the host (and then immediately disabling THP again, or by disabling THP permanently, as below, prior to rebooting).

      To ensure THP is permanently disabled, users can either:

      • put the above (root) command into their startup boot scripts (eg. /etc/rc.local), or
      • add transparent_hugepage=never to their Linux kernel commandline (eg. by editing the grub.conf file if they are using the GRUB bootloader), and then reboot. Note that some older Linux distributions have a bug where this setting is not effective, which means the above /etc/rc.local solution must be used instead (and the safest option may be to simply do both).


        Issue Links



              michael.paik Michael Paik
              kevin.pulo@mongodb.com Kevin Pulo
              Jonathan Dahl Jonathan Dahl
              2 Vote for this issue
              11 Start watching this issue


                6 years, 20 weeks, 2 days ago