[SERVER-40506] ftdc (hostInfo.num_cores,systemMetrics.cpu.num_cpus) to reflect vCPUs when containerized Created: 05/Apr/19 Updated: 05/Dec/23 Resolved: 05/Feb/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 4.3.4 |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Andrey Brindeyev | Assignee: | Billy Donahue |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Sprint: | Dev Tools 2020-02-10 | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||
| Description |
|
mongod and mongos processes should determine the correct number of vCPUs from POD's configuration when they executed in containerized environments. |
| Comments |
| Comment by James Broadhead (Inactive) [ 12/Feb/20 ] | ||||||||||||||||
|
billy.donahue if the server is able to only utilise resources inside the container, then I think it's appropriate to report those limits in FTDC. | ||||||||||||||||
| Comment by Billy Donahue [ 11/Feb/20 ] | ||||||||||||||||
|
Having finished this ticket, I'm now unsure if it was the right thing to have done. If FTDC is reporting on the health and limits of the machine on which we're running, then we should report whole-system stats. I'm coming to the understanding that this holistic data is something that makes FTDC useful for support troubleshooting. If FTDC is to report only the view from inside a container, then we should be reporting different stats as requested by this ticket. But which is it? | ||||||||||||||||
| Comment by Githook User [ 05/Feb/20 ] | ||||||||||||||||
|
Author: {'name': 'Billy Donahue', 'username': 'BillyDonahue', 'email': 'billy.donahue@mongodb.com'}Message: | ||||||||||||||||
| Comment by Billy Donahue [ 31/Jan/20 ] | ||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ] | ||||||||||||||||
|
Actually SERVER-36822 may be a better ticket for this. Feel free to close one or the other. EDIT: sorry, my mistake. That ticket is about adapting in systems where the number of CPUs can change dynamically. This ticket is probably better for covering correct reporting of container limits. | ||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 23/Oct/19 ] | ||||||||||||||||
|
The CPU numbers reported in ftdc (specifically hostInfo.num_cores and systemMetrics.cpu.num_cpus) use getNumCores and not getNumAvailableCores so they don't reflect container limits. | ||||||||||||||||
| Comment by Matt Lord (Inactive) [ 26/Apr/19 ] | ||||||||||||||||
|
Just noting that the underlying feature was added in 3.4+ inĀ | ||||||||||||||||
| Comment by Matt Lord (Inactive) [ 26/Apr/19 ] | ||||||||||||||||
|
I'm going to close this for now as mongod does take the available vCPU count (CPU hardware threads) for the process into account. To test and demonstrate that we're already doing this by taking cpu affinity into account: I first just added a new log message:
And then tested it this way:
As you can see, we correctly detected that there are only two available vCPUs. But there are two ways to go about setting this limit for a Docker container (see the Docker docs here):
As you can see, we're already working fine when the number of vCPUs has been set using --cpuset-cpus. The problem exists when you're attempting to use CPU time slices with --cpus, as then you're allocating a max percentage of the cpu cycles the process can use on the given cpu set (for example 25% cpu usage across the full vCPUs set). The time slice percentage for the process's current cgroup can be calculated from the value found at:
If it's support for CPU time slices that's needed, or something else, then we should open a more specifically targeted ticket for that work. Please let me know if you feel I made any errors here and we can re-open the ticket. Thanks! |