[DOCS-2131] Improve Linux transparent hugepages details in production notes Created: 22/Oct/13  Updated: 30/Oct/23  Resolved: 03/Apr/15

Status: Closed
Project: Documentation
Component/s: manual
Affects Version/s: None
Fix Version/s: Server_Docs_20231030

Type: Task Priority: Critical - P2
Reporter: Kevin Pulo Assignee: Michael Paik
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by DOCS-4864 Clearly document required MongoDB con... Closed
Related
related to SERVER-17418 After disabling transparent huge page... Closed
related to SERVER-7953 Warn on hugepages/THP Closed
is related to DOCS-5330 Using /etc/rc.local to disable THP re... Closed
is related to SERVER-17522 THP startup warnings can become stale Closed
Participants:
Days since reply: 7 years, 29 weeks ago

 Description   

The Linux Recommended Configuration section of the MongoDB production notes states

Do not use hugepages virtual memory pages as MongoDB performs better with normal virtual memory pages.

There are a few issues with this.

  1. It conflates "hugepages" and "transparent hugepages" (THP), which are related but not the same thing.
    1. Hugepages must be preallocated and require application-level support (which MongoDB does not have).
    2. Transparent hugepages (THP) are an operating system feature to automatically and transparently (to the application) use hugepages when possible.
  2. No information is given as to how users can:
    1. Which Linux distributions (and versions) have THP enabled by default.
    2. How users can check whether or not THP is enabled.
    3. How users can investigate potential performance problems arising from THP.
    4. How users can disable THP (temporarily and/or permanently).

Regarding the first point, it would be sufficient to adjust the sentence to read "Ensure that transparent hugepages are disabled, as MongoDB performs better with normal virtual memory pages" or some-such.

I believe that THP is only enabled by default in RHEL6 and SLES11 (and derivatives), and is disabled otherwise, but this might not be the case. If it is, this issue's relevance can be expected to increase over time as existing RHEL5 based systems are progressively upgraded to RHEL6.

To check if THP is enabled, users need to check the contents of /sys/kernel/mm/transparent_hugepage/enabled or /sys/kernel/mm/redhat_transparent_hugepage/enabled on Redhat-derived systems (ie. RHEL, CentOS, Amazon, etc). Some systems may have /sys/kernel/mm/transparent_hugepage as a symlink to /sys/kernel/mm/redhat_transparent_hugepage. Users could potentially be instructed to run

cat /sys/kernel/mm/*transparent_hugepage/enabled

to determine this in a Linux distro agnostic way. If neither of these files are present, then THP support is not compiled into the Linux kernel, and so is as good as disabled.

If THP is enabled, the output will look like:

[always] never

or on some systems:

[always] madvise never

If THP is disabled, the output will look like one of:

always [never]

always madvise [never]

always [madvise] never

To check possible performance problems arising from THP, users can check the khugepaged kernel thread process, as well as the contents of the other files in /sys/kernel/mm/*transparent_hugepage. In particular, any of the following are red flags:

  • enabled set to always
  • defrag set to always
  • khugepaged/defrag set to yes or 1
  • High values in any of khugepaged/full_scans, khugepaged/pages_collapsed or khugepaged/pages_to_scan
  • High cumulative cpu time for the khugepaged process
  • High cpu% for the khugepaged process
  • High or increasing value of AnonHugePages in /proc/meminfo
  • High or increasing value of thp_fault_alloc (or any other thp_* value) in /proc/vmstat

To disable THP temporarily (eg. to assess any possible impact), users can run as root:

for i in /sys/kernel/mm/*transparent_hugepage/enabled; do echo never > $i; done

or equivalently as a regular user:

echo never | sudo tee /sys/kernel/mm/*transparent_hugepage/enabled

This will stop khugepaged from performing any actions. Restarting mongod is not required for the new THP settings to take effect.

When disabling THP, any previously allocated transparent hugepages (reported by AnonHugePages in /proc/meminfo) will not be broken up. In most situations, this is unlikely to be a problem, but if it is suspected, they can be cleared by rebooting the host (and then immediately disabling THP again, or by disabling THP permanently, as below, prior to rebooting).

To ensure THP is permanently disabled, users can either:

  • put the above (root) command into their startup boot scripts (eg. /etc/rc.local), or
  • add transparent_hugepage=never to their Linux kernel commandline (eg. by editing the grub.conf file if they are using the GRUB bootloader), and then reboot. Note that some older Linux distributions have a bug where this setting is not effective, which means the above /etc/rc.local solution must be used instead (and the safest option may be to simply do both).


 Comments   
Comment by Emily Hall [ 27/Jul/16 ]

Closed for housekeeping on 7/27/2016 by Emily Hall.
If you require additional support, please open a new ticket for prioritization.
Thanks,
Emily

Comment by Githook User [ 20/Mar/15 ]

Author:

{u'username': u'mpaik', u'name': u'Michael Paik', u'email': u'michael.paik@10gen.com'}

Message: DOCS-2131 Production Notes Edits
Branch: v2.6
https://github.com/mongodb/docs/commit/f290feafbd8c239fc406b6fa40a5089684f4f726

Comment by Githook User [ 20/Mar/15 ]

Author:

{u'username': u'mpaik', u'name': u'Michael Paik', u'email': u'michael.paik@10gen.com'}

Message: DOCS-2131
Branch: v2.6
https://github.com/mongodb/docs/commit/00640f419e12bc3706f39f259edb8126df420daa

Comment by Githook User [ 18/Mar/15 ]

Author:

{u'username': u'mpaik', u'name': u'Michael Paik', u'email': u'michael.paik@10gen.com'}

Message: DOCS-2131

DOCS-2131 ProdNotes Edits
Branch: master
https://github.com/mongodb/docs/commit/c338cfe66491061252dc0dc1b042a918a7aa3c93

Comment by Githook User [ 18/Mar/15 ]

Author:

{u'username': u'mpaik', u'name': u'Michael Paik', u'email': u'michael.paik@10gen.com'}

Message: DOCS-2131

DOCS-2131 ProdNotes Edits
Branch: master
https://github.com/mongodb/docs/commit/c338cfe66491061252dc0dc1b042a918a7aa3c93

Comment by Kevin Pulo [ 05/Jun/14 ]

This comment expands on the rationale behind disabling THP when running MongoDB.

For background, the main design goals of THP are:

  • Easy to use
  • Reduce rate of TLB misses
  • Make TLB misses cheaper

The main problems with THP (in a MongoDB setting) are:

  • In the presence of memory pressure (very common with MongoDB), transparent hugepages can be automatically split so they are eligible to be swapped out (even though they won't be). In extreme cases, pages can be repeatedly split and coalesced many times, wasting large amounts of CPU time.
  • The potential win from TLB savings (when THP is enabled) is likely to be very small relative to the expenses incurred by page faults and/or network traffic, both of which are vastly more common with MongoDB. By contrast, the potential penalty from the overheads of running with THP may be very high.
Generated at Thu Feb 08 07:42:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.