[SERVER-40506] ftdc (hostInfo.num_cores,systemMetrics.cpu.num_cpus) to reflect vCPUs when containerized Created: 05/Apr/19  Updated: 05/Dec/23  Resolved: 05/Feb/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.3.4

Type: Improvement Priority: Minor - P4
Reporter: Andrey Brindeyev Assignee: Billy Donahue
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-22398 Add SystemInfo::numAvailableCores Closed
Duplicate
is duplicated by SERVER-36822 Report correct number of CPUs Backlog
Related
related to SERVER-83666 Improve clarity of number of CPUs met... Closed
Backwards Compatibility: Fully Compatible
Sprint: Dev Tools 2020-02-10
Participants:
Case:

 Description   

mongod and mongos processes should determine the correct number of vCPUs from POD's configuration when they executed in containerized environments.



 Comments   
Comment by James Broadhead (Inactive) [ 12/Feb/20 ]

billy.donahue if the server is able to only utilise resources inside the container, then I think it's appropriate to report those limits in FTDC.

Comment by Billy Donahue [ 11/Feb/20 ]

Having finished this ticket, I'm now unsure if it was the right thing to have done.
It depends on what the purpose of FTDC is.

If FTDC is reporting on the health and limits of the machine on which we're running, then we should report whole-system stats. I'm coming to the understanding that this holistic data is something that makes FTDC useful for support troubleshooting.

If FTDC is to report only the view from inside a container, then we should be reporting different stats as requested by this ticket. But which is it?

Comment by Githook User [ 05/Feb/20 ]

Author:

{'name': 'Billy Donahue', 'username': 'BillyDonahue', 'email': 'billy.donahue@mongodb.com'}

Message: SERVER-40506 hostInfo & FTDC to report container vCPU
Branch: master
https://github.com/mongodb/mongo/commit/746549116acd3139e64520159e8542387edce0b1

Comment by Billy Donahue [ 31/Jan/20 ]

https://mongodbcr.appspot.com/533390022

Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ]

Actually SERVER-36822 may be a better ticket for this. Feel free to close one or the other.

EDIT: sorry, my mistake. That ticket is about adapting in systems where the number of CPUs can change dynamically. This ticket is probably better for covering correct reporting of container limits.

Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ]

The CPU numbers reported in ftdc (specifically hostInfo.num_cores and systemMetrics.cpu.num_cpus) use getNumCores and not getNumAvailableCores so they don't reflect container limits.

Comment by Matt Lord (Inactive) [ 26/Apr/19 ]

Just noting that the underlying feature was added in 3.4+ inĀ SERVER-22398.

Comment by Matt Lord (Inactive) [ 26/Apr/19 ]

I'm going to close this for now as mongod does take the available vCPU count (CPU hardware threads) for the process into account. To test and demonstrate that we're already doing this by taking cpu affinity into account:

I first just added a new log message:

diff --git a/src/mongo/db/log_process_details.cpp b/src/mongo/db/log_process_details.cpp
index 9435fc2448..439f9662b6 100644
--- a/src/mongo/db/log_process_details.cpp
+++ b/src/mongo/db/log_process_details.cpp
@@ -58,6 +58,8 @@ void logProcessDetails() {
               << ProcessInfo::getSystemMemSizeMB() << " MB total system memory";
     }
 
+    log() << "Number of vCPUs available: " << ProcessInfo::getNumAvailableCores();
+
     printCommandLineOpts();
 }

And then tested it this way:

$ sudo docker run -it --rm -v `pwd`:`pwd` -w `pwd` -e LD_LIBRARY_PATH=./lib --cpuset-cpus=0-1 ubuntu:18.04 ./mongod --dbpath /tmp 2>&1 |
 grep "vCPU"                                                                                
2019-04-14T18:21:37.528+0000 I  CONTROL  [initandlisten] Number of vCPUs available: 2 

As you can see, we correctly detected that there are only two available vCPUs.

But there are two ways to go about setting this limit for a Docker container (see the Docker docs here):

  1. --cpus=<decimal> sets max cpu time slices across the set of hardware threads available
  2. --cpuset-cpus=<cpu set> uses scheduler affinity to limit the hardware threads available (tested above)

As you can see, we're already working fine when the number of vCPUs has been set using --cpuset-cpus. The problem exists when you're attempting to use CPU time slices with --cpus, as then you're allocating a max percentage of the cpu cycles the process can use on the given cpu set (for example 25% cpu usage across the full vCPUs set). The time slice percentage for the process's current cgroup can be calculated from the value found at:

/sys/fs/cgroup/cpu/cpu.shares

If it's support for CPU time slices that's needed, or something else, then we should open a more specifically targeted ticket for that work.

Please let me know if you feel I made any errors here and we can re-open the ticket. Thanks!

Generated at Thu Feb 08 04:55:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.