-
Type: Task
-
Resolution: Done
-
Priority: Critical - P2
-
Affects Version/s: None
-
Component/s: manual
-
Labels:None
The Linux Recommended Configuration section of the MongoDB production notes states
Do not use hugepages virtual memory pages as MongoDB performs better with normal virtual memory pages.
There are a few issues with this.
- It conflates "hugepages" and "transparent hugepages" (THP), which are related but not the same thing.
- Hugepages must be preallocated and require application-level support (which MongoDB does not have).
- Transparent hugepages (THP) are an operating system feature to automatically and transparently (to the application) use hugepages when possible.
- No information is given as to how users can:
- Which Linux distributions (and versions) have THP enabled by default.
- How users can check whether or not THP is enabled.
- How users can investigate potential performance problems arising from THP.
- How users can disable THP (temporarily and/or permanently).
Regarding the first point, it would be sufficient to adjust the sentence to read "Ensure that transparent hugepages are disabled, as MongoDB performs better with normal virtual memory pages" or some-such.
I believe that THP is only enabled by default in RHEL6 and SLES11 (and derivatives), and is disabled otherwise, but this might not be the case. If it is, this issue's relevance can be expected to increase over time as existing RHEL5 based systems are progressively upgraded to RHEL6.
To check if THP is enabled, users need to check the contents of /sys/kernel/mm/transparent_hugepage/enabled or /sys/kernel/mm/redhat_transparent_hugepage/enabled on Redhat-derived systems (ie. RHEL, CentOS, Amazon, etc). Some systems may have /sys/kernel/mm/transparent_hugepage as a symlink to /sys/kernel/mm/redhat_transparent_hugepage. Users could potentially be instructed to run
cat /sys/kernel/mm/*transparent_hugepage/enabled
to determine this in a Linux distro agnostic way. If neither of these files are present, then THP support is not compiled into the Linux kernel, and so is as good as disabled.
If THP is enabled, the output will look like:
[always] never
or on some systems:
[always] madvise never
If THP is disabled, the output will look like one of:
always [never]
always madvise [never]
always [madvise] never
To check possible performance problems arising from THP, users can check the khugepaged kernel thread process, as well as the contents of the other files in /sys/kernel/mm/*transparent_hugepage. In particular, any of the following are red flags:
- enabled set to always
- defrag set to always
- khugepaged/defrag set to yes or 1
- High values in any of khugepaged/full_scans, khugepaged/pages_collapsed or khugepaged/pages_to_scan
- High cumulative cpu time for the khugepaged process
- High cpu% for the khugepaged process
- High or increasing value of AnonHugePages in /proc/meminfo
- High or increasing value of thp_fault_alloc (or any other thp_* value) in /proc/vmstat
To disable THP temporarily (eg. to assess any possible impact), users can run as root:
for i in /sys/kernel/mm/*transparent_hugepage/enabled; do echo never > $i; done
or equivalently as a regular user:
echo never | sudo tee /sys/kernel/mm/*transparent_hugepage/enabled
This will stop khugepaged from performing any actions. Restarting mongod is not required for the new THP settings to take effect.
When disabling THP, any previously allocated transparent hugepages (reported by AnonHugePages in /proc/meminfo) will not be broken up. In most situations, this is unlikely to be a problem, but if it is suspected, they can be cleared by rebooting the host (and then immediately disabling THP again, or by disabling THP permanently, as below, prior to rebooting).
To ensure THP is permanently disabled, users can either:
- put the above (root) command into their startup boot scripts (eg. /etc/rc.local), or
- add transparent_hugepage=never to their Linux kernel commandline (eg. by editing the grub.conf file if they are using the GRUB bootloader), and then reboot. Note that some older Linux distributions have a bug where this setting is not effective, which means the above /etc/rc.local solution must be used instead (and the safest option may be to simply do both).
- is depended on by
-
DOCS-4864 Clearly document required MongoDB configuration in MMS OnPrem installation instructions
- Closed
- is related to
-
DOCS-5330 Using /etc/rc.local to disable THP results in spurious startup warning
- Closed
-
SERVER-17522 THP startup warnings can become stale
- Closed
- related to
-
SERVER-17418 After disabling transparent huge pages via rc.local mongodb shell warns about THP being enabled
- Closed
-
SERVER-7953 Warn on hugepages/THP
- Closed