[SERVER-40507] Track I/O stats in diagnostic data when mongod deployed in a POD using PersistentVolumeClaim Created: 05/Apr/19 Updated: 27/Oct/23 Resolved: 28/Feb/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andrey Brindeyev | Assignee: | Billy Donahue |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | kubernetes | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Sprint: | Dev Tools 2020-02-10, Dev Tools 2020-02-24, Dev Tools 2020-03-09 | ||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
At the moment the local SSD stats are recorded in FTDC while stats from PersistentVolumeClaim volumes are missing. |
| Comments |
| Comment by Billy Donahue [ 28/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Seems like we're reporting already, but need | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 28/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
That FTDC contains entries for xvdbb:
The difficulty is that nobody would know xvdbb is the /data mount point. Maybe we just need to include the mount table in the FTDC or the hostInfo so the FTDC disk stats can be interpreted properly. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Louis Plissonneau (Inactive) [ 28/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Here is the output of db.adminCommand({getDiagnosticData : 1})
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 28/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Louis gave me a way into his kube pod.
Explanation: I dig into mongod’s cmdline. Its arg3 is a mongod.conf file. We find the dbPath line in there. Find that dbPath in mount. Mount shows it under /dev/xvdbb but that doesn’t exist. What does exist is /sys/block/xvdbb so we look there and stat its device symlink, finding that it is a directory. This is sufficient criteria for the xvdbb to show up in FTDC. The "persistent volume" spec from which this device comes:
It's an EBS (ie "Elastic Block Store") which is attached to the container as a block device directly, nothing do with nfs or anything like that. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 13/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Transitioning to "blocked" while we gather specific requirements. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 11/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Is FTDC supposed to be reporting system-wide limits and stats or just container-imposed limits and stats? I feel like we aren't consistent about this and it's hard to write a comprehensive solution, I think it's because I'm not working from the context of an overarching epic, but just a few decoupled work items. It would be good to get more clarity on the expectations for these stats from the product side. One solution here, leaning more toward the whole-system direction, would be to simply report on ALL the block devices in the system. We are currently filtering the /proc/diskstats to show only "interesting" disks. We define a Linux block device to be "interesting" if it represents a physically connected device. We could conceivably solve this ticket by removing that filter and reporting about all block devices. If the Kubernetes Persistent Volume Claim bindings appear as all or part of some kind of block device, they'd turn up in the unfiltered /proc/diskstats. If they are implemented as some other kind of mount point (via fusefs or nfs tricks) that ISN'T backed by a block device they'd remain invisible. I'm not sure those would be visible even with cgroup blkio stats reads. They might show up as network traffic instead, for example. — I just wrote a similar statement in the closely related | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 02/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Okay, I'm looking at how lxcfs gathers its stats. It's all user-space. I don't understand it all yet but we could do what lxcfs does. $ cat /sys/fs/cgroup/blkio/blkio.io_merged $ cat /sys/fs/cgroup/blkio/blkio.io_service_bytes ETC.... | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Billy Donahue [ 01/Feb/20 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This is pretty challenging. What we do now in ftdc_system_stats_linux.cpp is iterate through all the block devices listed in /proc/diskinfo, I did find this FUSEfs module that replaces /proc/diskinfo (among other things) with Kubernetes-enhanced stats.
https://linuxcontainers.org/lxcfs/introduction/ https://dzone.com/articles/kubernetes-demystified-using-lxcfs-to-improve-cont It's possible the work required for this ticket is "tell people to install LXCFS"? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Brian Lane [ 03/Jun/19 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can platforms take a look at this one? |